General user interface gesture lexicon and grammar frameworks for multi-touch, high dimensional touch pad (HDTP), free-space camera, and other user interfaces

ABSTRACT

A method for a multi-touch gesture-based user interface wherein a plurality of gestemes are defined as functions of abstract space and time and further being primitive gesture segments that can be concatenated over time and space to construct gestures. Various distinct subset of the gestemes can be concatenated in space and time to construct a distinct gestures. Real-time multi-touch gesture-based information provided by user interface is processed to at least a recognized sequence of specific gestemes and that the sequence of gestemes that the user&#39;s execution a gesture has been completed. The specific gesture rendered by the user is recognized according to the sequence of gestemes. Many additional features are then provided from this foundation, including gesture grammars, structured-meaning gesture-lexicon, context, and the use of gesture prosody.

CROSS REFERENCE TO RELATED APPLICATIONS

Pursuant to 35 U.S.C. §119(e), this application claims benefit ofpriority from provisional patent application Ser. No. 61/449,923, filedMar. 7, 2011, and provisional patent application Ser. No. 61/482,606,filed May 4, 2011, the contents of each of which are hereby incorporatedby reference herein in their entirety.

COPYRIGHT & TRADEMARK NOTICES

A portion of the disclosure of this patent document may containmaterial, which is subject to copyright protection. Certain marksreferenced herein may be common law or registered trademarks of theapplicant, the assignee or third parties affiliated or unaffiliated withthe applicant or the assignee. Use of these marks is for providing anenabling disclosure by way of example and shall not be construed toexclusively limit the scope of the disclosed subject matter to materialassociated with such marks.

BACKGROUND OF THE INVENTION

The invention relates generally to the area of gesture-based userinterfaces, and more specifically to the creation of grammars forgesture-based user interfaces, particularly in the context oftouch-based user interfaces.

Until recent years the dominant form of Graphical User Interface (GUI)model for general-purpose computers has been (initially) the DirectManipulation and Desktop Metaphor (see for examplehttp://en.wikipedia.org/wiki/Direct_manipulation), often attributed toB. Shneiderman in 1983[1], and later their arguable descendent WIMP(“Window, Icon, Menu, Pointer/Pointing/Pull-Down/Pop-up”) GUI (see forexamplehttp://en.wikipedia.org/wiki/History_of_the_graphical_user_interface andhttp://en.wikipedia.org/wiki/WIMP_(computing)). Many additional userinterface mechanisms have been explored, and many of these (for example,speech recognition) map directly into the Direct Manipulation andDesktop Metaphor paradigm. The pointing devices employed notably includenot only the computer mouse but a number of surrogate forms emulatingthe mouse metaphor, namely various trackballs, keyboard-sticks,touch-screens, and touchpads (including the KoalaPad™ in 1984—see forexample http://en.wikipedia.org/wiki/Koala_Pad). These touch-basedcomputer interfaces (touch-screens and touchpads) indeed operated asmere stand-in emulations of computer mouse functionality.

It is noted that, prior to computer touch-screens and touchpads variouselevator, machine, and appliance controls from the 1950's (and likelyearlier) included touch-operated on-off switches, and various 1970'smusic synthesizers included touch-keyboards and one-dimensional touch“ribbon controllers.”

Work on more sophisticated touch-based computer and control interfacesthat accommodate and utilize touch-based gestures has a long history,some of it widely recognized (for examplehttp:/www.billbuxton.com/multi-touchOverview.htm) and less well-knownsuch as the High Dimensional Touch Pad (HDTP) technology represented forexample by (1999 priority date) U.S. Pat. No. 6,570,078, U.S. patentapplication Ser. No. 11/761,978, U.S. patent application Ser. No.12/418,605, and some at least two dozen other related pending patentapplications. The most well-known work is that of Wayne Westerman andhis thesis professor John Elias. The approach that work took totouch-based gestures has since been incorporated into in a large numberof Apple™ products, and subsequently widely adopted by large a number ofother handheld, tablet, laptop, and other computing-based devices madeby many product manufacturers.

Within this period of time there was a considerable amount of work andproduct relating to pen/stylus-based handwriting interfaces (see forexample http://en.wikipedia.org/wiki/Pen_computing), some including afew early gesture capabilities(http://en.wikipedia.org/wiki/Pen_computing#Gesture_recognition).

More recently video-camera-based free-space hand-gesture input haveappeared, It is noted that (1999 priority date) U.S. Pat. No. 6,570,078teaches use of a video camera as an input device to deliver HDTPcapabiities extended to free-space hand-gesture input.

Although the widely adopted approach to gesture-based multi-touch userinterfaces developed by Westerman and Apple has become pervasive andextends the WIMP GUI operations to include a number of allegedly “new”metaphor-based specialty operations (such as “swipe,” “stretch,”“pinch,” “rotate,” etc), that approach is hardly the last word intouch-based user interfaces. The HDTP approach to touch-based userinterfaces, represented for example by represented for example by U.S.Pat. No. 6,570,078, U.S. patent application Ser. No. 11/761,978, U.S.patent application Ser. No. 12/418,605, provides a framework thatincludes or supports today's widely adopted gesture-based multi-touchuser interface features and further supports a wide range of additionalcapabilities which transcend and depart from today's widely adoptedgesture-based multi-touch user interfaces.

A first aspect of the HDTP approach includes the capability for derivingmore than the two-dimensional ‘continuous-adjustment’ user inputs thanare provided by today's widely adopted gesture-based multi-touch userinterface “geometric location” operations (such as X-Y location, “flick”X-Y location-change velocity, “flick” X-Y location-change angle). Forexample the HDTP approach to touch-based user interfaces can provideadditional ‘continuous-adjustment’ user inputs such as:

-   -   Yaw-angle of a contacting finger, thumb, palm, wrist, etc.;    -   Roll-angle of a contacting finger, thumb, palm, wrist, etc.;    -   Pitch-angle of a contacting finger, thumb, palm, wrist, etc.;    -   Downward pressure of a contacting finger;    -   Spread angle between each pair of contacting finger(s), thumb,        palm, wrist, etc.;    -   Differences in X location between each pair of contacting        finger(s), thumb, palm, wrist, etc.;    -   Differences in Y location between each pair of contacting        finger(s), thumb, palm, wrist, etc.;    -   Differences in downward pressure between each pair of contacting        finger(s), thumb, palm, wrist, etc.;    -   Rates-of-change for the above.

These additional capabilities widely expand the number and types ofgestural, geometric, and spatial-operation metaphors that can beprovided by touch interfaces. Further, various types of conditionaltests may be imposed on these additional ‘continuous-adjustment’ inputs,permitting productions of and associations with symbols, domains,modalities, etc.

Today's widely adopted gesture-based multi-touch user interfacesrecognize the number of multiple-touch contacts with the touch interfacesurface. A second aspect of the HDTP approach to touch-based userinterfaces are additional ‘shape’ user input recognitions distinguishingamong parts of the hand such as:

-   -   Finger-tip;    -   Finger-joint;    -   Flat-finger;    -   Thumb;    -   Cuff;    -   Wrist;    -   Palm;    -   Left-hand;    -   Right-hand.

Today's widely adopted gesture-based multi-touch user interfacesrecognize individual isolated gestures. A third aspect of the HDTPapproach to touch-based user interfaces can provide yet other additionalfeatures such as:

-   -   Compound touch gestures;    -   Attributes of individual component elements comprised by a        gesture such as:        -   Order of individual component element rendering;        -   Relative location of individual component element rendering;        -   Embellishment in individual component element rendering            (angle of rendering, initiating curve, terminating curve,            intra-rendering curve, rates of rendering aspects, etc.);    -   Connected gestures;    -   Context-based interpretation/action/semantics;    -   Inheritance-based interpretation/action/semantics;    -   Syntactic grammars.        The present patent application, along with other associated        co-pending U.S. patent cited herein, directs further attention        to these topics, both in the context of HDTP technology as well        as other user interface technologies including:    -   Simple touch user interface systems found in handheld devices,        laptops, and other mobile devices    -   Video camera-based free-space gesture user interface systems

In the case of the HDTP approach to touch-based user interfaces, theseprovide the basis for

-   -   (1) a dense, intermixed quantity-rich/symbol-rich/metaphor-rich        information flux capable of significant human-machine        information-transfer rates and    -   (2) an unprecedented range of natural gestural metaphor support.        The latter (1) and its synergy with the former (2) is especially        noteworthy, emphasized the quote from the recent cover story in        the February 2011 Communications of the ACM [2]:    -   “Gestures are useful for computer interaction since they are the        most primary and expressive form of human communication.”

The next-generation user interface work in academia, as well as in videogames, however, is now directing attention to video-camera-basedfree-space gesture input, owing great debts to the pioneeringexperiential/installation/performance-art-oriented real-time video-basedcomputer control work of Myron Kruger. These camera-based free-spacegesture input user interfaces will be providing a range of possibilitiescomprising, at least tabula rasa, ranges and possibilities not unlikethose provided by the HDTP approach to touch-based user interfaces. (Infact (1999 priority date) U.S. Pat. No. 6,570,078, U.S. patentapplication Ser. No. 11/761,978 teach use of one or more video camerasas alternative input sensors to HDTP processing so as to respond tofree-space hand gestures.)

However, it is not at this time clear whether the camera-basedfree-space gesture input user interface community will see theseopportunities or simply incrementally adapt and build on WIMPframeworks, the Westerman/Apple approach, 3D extrapolations of desktops,etc. Additionally, these camera-based free-space gesture input userinterface approaches have their own usage challenges (not the least ofwhich including arm fatigue, input on/off detection (“Midas Touchproblem”) and computation challenges if trying to adopt rich-semanticinputs (for example, recognitions of ASL and other sign languagesremains computationally out or reach even well-funded research labsloaded with computers [2]).

It is believed this effort, in addition to the role it provides tocontemporary touch interfaces and HDTP technology, could deliverpotential utility to next-generation touch interfaces and provide aframework and an example perhaps of possible value to the camera-basedfree-space gesture input user interface community as the possibilitiesand opportunities for camera-based free-space gesture input userinterface technology and its applications are explored, developed, andformalized.

SUMMARY OF THE INVENTION

For purposes of summarizing, certain aspects, advantages, and novelfeatures are described herein. Not all such advantages may be achievedin accordance with any one particular embodiment. Thus, the disclosedsubject matter may be embodied or carried out in a manner that achievesor optimizes one advantage or group of advantages without achieving alladvantages as may be taught or suggested herein.

In an aspect of the invention, a method is provided for a multi-touchgesture-based user interface wherein a plurality of gestemes are definedas functions of abstract space and time and further being primitivegesture segments that can be concatenated over time and space toconstruct gestures. Various distinct subset of the gestemes can beconcatenated in space and time to construct a distinct gestures.

In another aspect of the invention, real-time multi-touch gesture-basedinformation provided by user interface is processed to at least arecognized sequence of specific gestemes and that the sequence ofgestemes that the user's execution a gesture has been completed.

In another aspect of the invention, the specific gesture rendered by theuser is recognized according to the sequence of gestemes.

In another aspect of the invention, many additional features areprovided from this foundation.

In another aspect of the invention, gesture grammars are provided.

In another aspect of the invention, structured-meaning gesture-lexiconframeworks are provided.

In another aspect of the invention, gesture context frameworks areprovided.

In another aspect of the invention, the use of gesture prosody isprovided.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features and advantages of the presentinvention will become more apparent upon consideration of the followingdescription of preferred embodiments taken in conjunction with theaccompanying drawing figures, wherein:

FIG. 1 depicts a representation of how the imposition of selectedwell-thought-through structures on computing hardware and softwaretechnologies has greatly facilitated the development of computingtechnology.

FIG. 2 depicts a representation of the tensions among maximizing theinformation rate of communication from the human to the machine,maximizing the cognitive ease in using the user interface arrangement,and maximizing the physical ease using the user interface arrangement

FIG. 3 depicts a representation of example relationships of traditionalwriting, gesture, and speech with time, space, direct marks, andindirect action.

FIG. 4a and FIG. 4b (adapted from [3]) illustrates an example set offour primitive handwriting segment shapes that could be used ascomponents for representation of cursive-style handwrittenEnglish-alphabet letters.

FIG. 5 (also adapted from [3]) illustrates an example decomposition ofcursive-style handwritten English-alphabet letters in terms of theexample set of eighteen primitive handwriting “graphemes” depicted inFIG. 4 a.

FIG. 6 depicts a representation of a general user interface arrangementrelevant to the present invention.

FIG. 7a through FIG. 7c depict representations of an example touch-basedsingle-finger “finger-flick” gesture, wherein a finger makes physicalcontact begins in a first (initiating) location on a touch surface, andmoves remaining in contact with the touch surface to a second(terminating) location roughly along a straight-line path within apredefined minimum interval of time.

FIG. 8a through FIG. 8c depict representations of an example touch-basedsingle-finger hook-shaped gesture, wherein a finger makes physicalcontact begins in a first (initiating) location on a touch surface, andmoves remaining in contact with the touch surface along hook-shaped pathto a second (terminating) location within a predefined minimum intervalof time.

FIG. 9 depicts an example signal-space representation of thesingle-finger “finger-flick” gesture represented by FIG. 7a through FIG.7c , wherein a signal-space trajectory starts in a first (initiating)signal-space location and changes values to a second (terminating)signal-space location within a predefined minimum interval of time.

FIG. 10 depicts an example signal-space representation of thesingle-finger hook-shaped gesture represented by FIG. 8a through FIG. 8c, wherein a signal-space trajectory starts in a first (initiating)signal-space location and changes values to a second (terminating)signal-space location within a predefined minimum interval of time.

FIG. 11 depicts an example symbol generation arrangement for generatinga sequence of symbols from (corrected, refined, raw, adapted,renormalized, etc.) real-time measured parameters values provided byother portions of an HDTP system.

FIG. 12 depicts a modification of the exemplary arrangement of FIG. 11wherein symbol can be generated only under the control of a clock orsampling command, clock signal, event signal, or other symbol generationcommand.

FIG. 13, adapted from U.S. patent application Ser. No. 12/418,605,depicts a representation of an example symbol generation arrangement.

FIG. 14 depicts such a conditional test for a single parameter or ratevalue q in terms of a mathematical graph, separating the full range of qinto three distinct regions.

FIG. 15a and FIG. 15b depict a representation of a conditional test fora two values (parameter and/or rate) in terms of a mathematical graph,separating the full range of each of the two values into three regions.

FIG. 16a and FIG. 16b depict a representation of a conditional test fora two values (parameter and/or rate) in terms of a mathematical graph,separating the full range of each of the three values into threeregions.

FIG. 17 a representation of an intrinsic metaphor applied to a touchsensor that senses touch attributes, and these being directed to animposed metaphor causing an application response to be invoked on anassociated application.

FIG. 18 depicts a representation of a sequence of symbols can bedirected to a state machine so as to produce other symbols that serve asinterpretations of one or more possible symbol sequences.

FIG. 19 depicts a representation of a variation on FIG. 18 wherein oneor more symbols may be designated the meaning of an “Enter” key,permitting for sampling one or more varying parameter, rate, and/orsymbol values and holding the value(s) until, for example, another“Enter” event, thus producing sustained values.

FIG. 20 depicts a representation of further processing opportunitiessupporting a full range of postures, gestures, real-time parameterextractions, and information needed for implementations of gesturegrammars.

FIG. 21 and FIG. 22 depict representations of one or more symbols may bedesignated as setting a context for interpretation or operation and thuscontrol mapping and/or assignment operations on parameter, rate, and/orsymbol values, and further depict representations of context-orientedand context-free production of parameter, rate, and symbol values.

FIG. 23 depicts an example representation of a predefined gesturecomprised by a specific sequence of three other gestures.

FIG. 24 depicts an example representation of a predefined gesturecomprised by a sequence of five recognized gestemes.

FIG. 25 depicts a representation of a layered and multiple-channelmetaphor wherein the {x,y} location coordinates represent the locationof a first point in a first geometric plane, and the {roll,pitch} anglecoordinates are viewed as determining a second independently adjustedpoint on a second geometric plane.

FIG. 26 depicts a representation of the relations between gestureaffixes and interrupted gesture executions. Interrupted gestures canalso be more broadly supported by the present invention so as addresscovering non-affix cases.

FIG. 27a through FIG. 27j depict an example representation of theexecution of a first example predefined gesture that is begun (FIG. 27a) and interrupted (FIG. 27b and FIG. 27c ), the full execution of anexample second predefined gesture (FIG. 27d , FIG. 27e , FIG. 27f , andFIG. 27g ), and the resumed and completed execution of the firstpredefined gesture (FIG. 27h , FIG. 27i , and FIG. 27j ).

FIG. 28a through FIG. 28j depict a variation on the example of FIG. 27athrough FIG. 27j wherein the lift-off events depicted by FIG. 27c , FIG.27g , and FIG. 27j are replaced with the pause events depicted in FIG.28c with FIG. 28d , FIG. 28g with FIG. 28h , and in FIG. 28j . FIG. 28ais a repeat of FIG. 27a . FIG. 28b is a repeat of FIG. 27b . FIG. 28e isa repeat of FIG. 27e . FIG. 28f is a repeat of FIG. 27f . FIG. 28i is arepeat of FIG. 27 i.

FIG. 29a through FIG. 29f depict a variation on the example of FIG. 27athrough FIG. 27j wherein the lift-off events associated FIG. 27c , FIG.27g , and FIG. 27j are omitted altogether and semantic restrictions ongesteme sequences can be used to signify the completion of the secondgesture and the prompt for the completion of the first gesture.

FIG. 30 depicts a representation of some correspondences among gestures,gestemes, and the abstract linguistics concepts of morphemes, words, andsentences.

FIG. 31a through FIG. 31d depict representations of finer detail usefulin employing additional aspects of traditional linguistics such as nounphrases, verb phrases, and clauses as is useful for grammaticalstructure, analysis, and semantic interpretation.

FIG. 32a through FIG. 32d and FIG. 33a through FIG. 33f depictrepresentations of sequentially-layered execution of tactile gesturescan be used to keep a context throughout a sequence of gestures.

FIG. 34 depicts a representation of an example syntactic and/or semantichierarchy integrating the concepts developed thus far.

FIG. 35 depicts a representation of an example of two or morealternative gesture sequence expressions to convey the same meaning.

FIG. 36 depicts a representation of an example of a Unix™ Pipestandard-input/standard-output chain.

FIG. 37 depicts a representation of an example using intra-gestureprosody as a means of implementing both pipes and other associationsand/or data flo connections.

FIG. 38 depicts a composite view of some of the key the informationflows supported by the construction provided thus far.

FIG. 39a though FIG. 39c depict representations of aspects of a verysimple example grammar that can be used for rapid control of CAD ordrawing software.

FIG. 40 depicts how the simple example grammar can be used to control aCAD or drawing program.

DETAILED DESCRIPTION

In the following description, reference is made to the accompanyingdrawing figures which form a part hereof, and which show by way ofillustration specific embodiments of the invention. It is to beunderstood by those of ordinary skill in this technological field thatother embodiments may be utilized, and structural, electrical, as wellas procedural changes may be made without departing from the scope ofthe present invention.

In the following description, numerous specific details are set forth toprovide a thorough description of various embodiments. Certainembodiments may be practiced without these specific details or with somevariations in detail. In some instances, certain features are describedin less detail so as not to obscure other aspects. The level of detailassociated with each of the elements or features should not be construedto qualify the novelty or importance of one feature over the others.

1. Goal of Applicable and Enabling Structure

The imposing of a structure can be confining or empowering (and isusually to some extent both). For example, a large collection of digitallogic chips and analog electronic components can be used inunsophisticated ways to create a large number of scattered devices orprojects delivering dispersed and perhaps immense squandering ofresource, time, and opportunity. An example of a more sophisticated useof the large collection of digital logic chips and analog electroniccomponents can be to assemble a particular large-scale high-performancededicated-purpose device (for example, a video processor such as ahardware codec). The utility of the resulting device could be limited byany number of aspects, including being unable to include or work withnew innovations, the fickle evolution of video compression standards anduse of video communications by the user, etc. Another example of a moresophisticated use of the large collection of digital logic chips andanalog electronic components, however, is the creation of ageneral-purpose computing platform that could be used for a wide rangeof software and thus supporting a large number of valuable applicationsand able to maintain relevance over a range of evolutionary approaches.

In the case of computing hardware and software technologies, theimposition of selected well-thought-through structures has greatlyfacilitated the development of computing technology. As described inFIG. 1:

-   -   A widely ranging collection of hardware technology components        were advantageously structured to create computer platforms        providing structured environment for executing algorithm        instructions    -   A widely ranging collection of algorithm instructions were        advantageously structured to create operating system platforms        providing structured environment for executing software    -   A widely ranging collection of operating system components were        advantageously structured to create language platforms providing        structure environment for implementing software technologies;    -   A widely ranging collection of software technology components        were advantageously structured to create programming paradigms        providing structured environment for implementing application        programs        If the selected structures were not well-thought-through or not        available at all, it would have been essentially impossible for        computing hardware and software technologies to have progressed        to the level that they have.

It is the latter example of imposing selected well-thought-throughstructures that is the goal of the proposed lexicon and grammarconstruction and formalism for gestures—sought is a conceptual,software, and technical ‘platform’ for tactile user interface lexiconand grammar frameworks that could be used for a wide range ofconfigurations and thus supporting a large number of valuableapplications and able to maintain relevance over a range of evolutionaryapproaches. That is, in the analogy, sought is a structure imposed onthe analogous large collection of digital logic chips and analogelectronic components (analogous to the capabilities of touchinterfaces, particularly the HDTP approach to them) to built ananalogous flexible general-purpose computer (analogous to theconstruction of formalisms for tactile user interface lexicon andgrammar frameworks) that supports a large number of valuableapplications and able to maintain relevance over a range of evolutionaryapproaches. One cannot have a flexible general-purpose computer withoutimposing structure on the collection of components, or by imposing anunsophisticated, overly-limiting or overly-specialized structure on thecollection of components.

Ultimately the goal of command user interface arrangement is to balancethe tensions among maximizing the information rate of communication fromthe human to the machine, maximizing the cognitive ease in using theuser interface arrangement, and maximizing the physical ease using theuser interface arrangement. These three goals are not always in strictopposition but typically involve some differences hence resulting intradeoffs as suggested in FIG. 2.

Adoptions and adaptations of effective preceding approaches, leavingbehind what is not relevant and adding new things where advantageous, isexactly the process Thomas Kuhn spelled out in his work on the structureof scientific revolutions—the approach presented here shall draw fromknown user interfaces, traditional linguistics, temporal logic, andother established thought in synergistic leverage to the additionallyformalize the range and engineering of the capabilities provided by theexample of the HDTP approach to touch-based user interfaces. To begin,some adoptions and adaptations of traditional linguistics are employed.At a high level the goal is to achieve a high-performance user interfaceleveraging inherent, intuitive, and metaphorical aspects of language, soseeking utility from within selected aspects of traditional linguisticstheory.

2. Use of a Linguistics Framework

There are a number of more detailed reasons to engage the framework oftraditional linguistics, among these including that many of the conceptshave already been worked out, widely-accepted terminologies have alreadybeen established, and these concepts and terms provide a basis fordrawing on the expertise of contemporarily linguists. Further,traditional generative linguistics programs, for example thoseinfluenced by Chomsky, Jackendoff, and many notable others appeal to atheme of there being a set of underlying human language capabilitieswhich can be approached and approximated by various models (ExtendedStandard Theory, Y-Shape Models, Principles and Parameters, Governmentand Binding, etc.). Additionally, the goals sought by the charters ofNatural Language and Universal Grammar offer additional resources, andnumerous other formalisms (such as that of morphemes, syntacticstructure, lexicon, writing systems, etc.; even phonetics) provide agood setting and collection of resources from which to begin thisproject. In particular, as an initial foundation, the follow notionswill be employed (quick references to wiki summaries are provided):

-   -   Morphemes—http://en.wikipedia.org/wiki/Morpheme;    -   Language morphology frameworks using        morphemes—http://en.wikipedia.org/wiki/Morphology (linguistics):        -   Analytic language,        -   Agglutinative language,        -   Fusion language,        -   Polysynthetic language;    -   Phonemes/graphemes (by        analogy)—http://en.wikipedia.org/wiki/Phoneme,        http://en.wikipedia.org/wiki/Grapheme;    -   Orthography/writing        systems—http://en.wikipedia.org/wiki/Orthography,        http://en.wikipedia.org/wiki/Writing_system;    -   Phonetic onomatopoeia (by        analogy)—http://en.wikipedia.org/wiki/Onomatopoeia;    -   Logography—http://en.wikipedia.org/wiki/Logography;    -   Clitics (particular        endoclitics)—http://en.wikipedia.org/wiki/Clitic;    -   Lexicon—http://en.wikipedia.org/wiki/Lexicon;    -   Punctuation—http://en.wikipedia.org/wiki/Punctuation;    -   Prosody—http://en.wikipedia.org/wiki/Prosody_(linguistics);    -   Syntactic analysis/parsing—http://en.wikipedia.org/wiki/Parsing;    -   Lexical categories—http://en.wikipedia.org/wiki/Lexical        category;    -   Phrases, clauses, and sentences;    -   Syntax and sentence        grammar—http://en.wikipedia.org/wiki/Grammar;    -   Context.

However, the capabilities of touch interfaces, at least as provide bythe HDTP approach to touch-based user interfaces, can include featuresinvolving other types of formalisms, for example:

-   -   adaptations of temporal logic (as will be explained);    -   standard-input/standard-output;    -   multi-threaded/parallelism.

So with this foreground preparation in place, the construction offormalisms for tactile user interface lexicon and grammar frameworkswill begin.

3. Gesture Structure, Constituents, Execution, and Machine Acquisition

A tactile gesture is a bit like traditional writing in some ways anddiffers from writing in other ways. Like traditional writing a tactilegesture involves actions of user-initiated contact with a surface and isrendered over a (potentially reusable) region of physical surface area.The term “execution” will be used to denote the rendering of a tactilegesture by a user via touch actions made on a touch interface surface.

In various implementations the execution of a tactile gesture by a usermay (like traditional writing) or may not (unlike writing) be echoed byvisible indication (for example a direct mark on the screen). In variousimplementations the symbol execution of a tactile gesture by a user maycomprise spatially isolated areas of execution (in analogy with thedrawing of block letters in traditional writing) or may comprisespatially isolated areas of symbol execution (in analogy with thedrawing of sequences of cursive or other curve-connected/line-connectedletters in traditional writing).

However, unlike traditional writing, a tactile gesture can includeprovisions to capture temporal aspects of its execution (for example thespeed in which it is enacted, the order in which touch motionscomprising the gesture are made, etc.). Also unlike traditional writing,the result of a tactile gesture can include a visually-apparent indirectaction displayed on a screen responsive to a meaning or metaphorassociated with the tactile gesture. In a way, these aspects are a bitlike speech or a speech interface to a computer—time is used rather thanspace for the rendering/execution, and the (visual) response (of amachine) can be one of an associated meaning.

FIG. 3 illustrates these example relationships of traditional writing,gesture, and speech with time, space, direct marks, and indirect action.Of course it is likely possible to construct or envision possible speechand writing systems that defy, extend, or transcend the relationshipsdepicted in FIG. 2, but for the moment with no ill-will orlimited-thinking intended these will, at least for now, be regarded asfringe cases with respect to the gesture lexicon and graphics frameworkpresented herein.

3.1 Phoneme, Grapheme, “Gesteme”

Like traditional writing and speech, tactile gestures can be comprisedof one or more constituent “atomic” elements. In the formal linguisticsof speech, these constituent “atomic” elements are known as phonemes. Inthe formal linguistics of traditional writing, the constituent “atomic”elements are termed graphemes (see for examplehttp://en.wikipedia.org/wiki/Grapheme).

Accordingly, in this construction the one or more constituent “atomic”elements of gestures will be called “gestemes;” examples includeisolated stroke lines, isolated curves, etc. For example, a gesture thatis spatially rendered by tracing out an “X” or “+” on a touch surfacewould (at least most naturally) comprise an action comprising two strokelines. Gesteme-based gesture structuring, recognition, processing arefurther treated in co-pending U.S. Patent Application 61/567,626.

In traditional (at least Western) writing, the order in which suchstrokes are rendered by the user, the time it takes to render eachstroke (“gesteme”), and the time between making the two strokes, andanything else that is done in a different spatial area (such as drawinganother letter) between making the two strokes are all immaterial as theinformation is conveyed by the completed “X” or “+” marking left behindafter the execution. The HDTP approach to touch-based user interfaces,however, allows for use of:

-   -   the time it takes to render each gesteme;    -   the time between rendering a pair of gestemes;    -   anything else that is done in a different spatial area (such as        the drawing of another symbol) between rendering a pair of        gestemes.

3.1.1 Relating Gestemes to Example “Graphemes” for RepresentingCursive-Style Handwritten English-Alphabet Letters

As discussed above in conjunction with FIG. 3, gestures have someattributes that are similar to speech and other attributed that aresimilar to writing. Thus it would be expected that gestemes would havesome attributes of graphemes.

Although there are other references to draw from regarding graphemes,FIG. 4a , adapted from a 1961 paper by M. Eden [3], illustrates anexample set of four primitive handwriting segment shapes that could beused as components for representation of cursive-style handwrittenEnglish-alphabet letters. FIG. 4b , also adapted from [3], illustratesan example an example set of eighteen primitive handwriting “graphemes”created from various translations and mirror-symmetry transformations ofthe example set of four primitive handwriting segment shapes depicted inFIG. 4 a.

FIG. 5, also adapted from [3], illustrates an example decomposition ofcursive-style handwritten English-alphabet letters in terms of theexample set of eighteen primitive handwriting “graphemes” depicted inFIG. 4a . In this example (Eden) system, the simultaneous presence ofspecific combinations of the eighteen primitive handwriting “graphemes”signifies a specific cursive-style handwritten English-alphabet letter.

FIG. 3 illustrates an example comparison of gestures with writing andspeech. Speech is rendered over time while writing is rendered overspace. Gestures have aspects of both writing and speech, for examplebeing rendered over space and over time. In relating this to the exampleprovided in FIG. 5, the example (Eden) system employs simplecombinational logic operations of the truth-values of the presence ofthe graphemes of FIG. 4b . In general (and in contrast), a gesture willreplace the simple combinational logic operations on the presence ofspecific graphemes used in writing with more complex “temporal logic”operations on the presence of specific graphemes. However, the temporalaspect of a rendered gesture can rightfully be included in the structureof primitive elements of gestures, as considered below and elsewhereherein.

3.1.2 Relating Gestemes to Phonemes: Gesteme Delineation within aGesture

As discussed above in conjunction with FIG. 3, gestures have someattributes that are similar to speech and other attributed that aresimilar to writing. Thus it would be expected that gestemes would havesome attributes of phonemes.

The following analogies with the traditionally considered phonemes ofspoken language can provide useful perspective on defining gestemedelineation within a gesture.

First, in analogously relating a gesteme to a phoneme comprisingbeginning an ending consonants surrounding a (mono)vowel, diphthong, ormore general form of gliding vowel:

Consonant-Enveloped Phoneme Gesteme Start Starting consonant Startingmotion and/or position Internal (mono)vowel, diphthong, Dynamic activityor gliding vowel End Ending consonant Ending motion and/or position

Second, in analogously relating a gesteme to a phoneme comprising abeginning consonant followed by a (mono)vowel, diphthong, or moregeneral form of gliding vowel:

Consonant-Leading Phoneme Gesteme Start Starting consonant Startingmotion and/or position Internal (mono)vowel, diphthong, Dynamic activityfurther comprising and End or gliding vowel the ending motion and/orposition

Third, in analogously relating a gesteme to a phoneme comprising a(mono)vowel, diphthong, or more general form of gliding vowel followedby an ending consonant:

Consonant-Consluding Phoneme Gesteme Start (mono)vowel, Starting motionand/or position further and diphthong, or gliding comprising continuingdynamic activity Internal vowel End Ending consonant Ending motionand/or position

Forth, in analogously relating a gesteme to a phoneme comprising only a(mono)vowel, diphthong, or more general form of gliding vowel:

Consonant-Free Phoneme Gesteme Start, (mono)vowel, diphthong, Startingmotion and position also Internal, or gliding vowel comprisingcontinuing dynamic and End activity

Of these four analogies, the first or fourth would typically provide anadequate framework for general use.

3.2 Gestures

In the construction of the formalism, a gesture may be equated to therole of a word or word group or compound work acting as a word. Thisapproach will be used for the moment, but with the incorporation ofadditional aspects of gesture rendering the linguistic domain andlinguistic function of a gesture can be expanded to include entiremulti-element noun phases, verb phrases, etc. (as will be considered inlater sections of this document pertaining to grammar).

3.2.1 Gesture Composition from Gestemes

The HDTP approach to touch-based user interfaces also allows for asingle gestemes to be used as a gesture. However, the HDTP approach totouch-based user interfaces more commonly allows for the concatenationof two or more gestemes to be sequentially rendered (within thedelimiters of a gesture) to form a gesture.

In some cases, gestemes may be defined in such a way that naturaljoining is readily possible for all, most, or some combinations ofconsecutive pairs of gestemes. In some cases, some form of shortening orbridging may be used to introduce economy or provide feasibility in thejoining pairs of consecutive gestemes.

3.2.2 Gesteme Sequencing within the Rendering of a Gesture

The HDTP approach to touch-based user interfaces also allows for thereto be additional content to be imposed into/onto the individual gestemesused to render even such simple “X” or “+” gestures. For example:

-   -   The order in which the user renders the two strokes can be        ignored, or could instead be used to convey meaning, function,        association, etc.;    -   The absolute or relative time the user takes to render each        stroke can be ignored, or could instead be used to convey a        quantity, meaning, function, association, etc.;    -   The absolute or relative time the user takes between the        rendering of each stroke can be ignored, or could instead be        used to convey a quantity, meaning, function, association, etc.    -   An action (for example, a tactile action) taken by the user        between the rendering of each stroke can be ignored, or could        instead be used to convey a quantity, meaning, function,        association, etc.

The temporal aspects involved in each of the above examples brings inthe need for an adapted temporal logic aspect to formalisms for tactileuser interface lexicon and grammar frameworks should these temporalaspects be incorporated. Depending upon the usage, the temporal logicaspect framework would be used to either distinguish or neglect therendering order of individual gestemes comprising a gesture.

3.2.3 Delimiters for Individual Gestures

In the rendering of speech, delimiting between individual words isperformed through use of one or more of the following:

-   -   Prosody:        -   Temporal pause;        -   Changes in rhythm;        -   Changes in stress;        -   Changes in intonation.    -   Lexigraphics (an individual word is unambiguously recognized,        and the recognition event invokes a delineating demarcation        between the recognized word and the next word to follow).

In the rendering of traditional writing, delimiting between individualwords is performed via gaps (blank spaces roughly the space of acharacter).

The HDTP approach to touch-based user interfaces provides for delimitingbetween individual temporal tactile gestures via at least thesemechanisms:

-   -   Time separation between individual tactile gestures;    -   Distance separation between individual tactile gestures;    -   For joined strings of individual tactile gestures:        -   Temporal pause separation;        -   Logographically separation;        -   Lexigraphically separation (an individual tactile gesture is            unambiguously recognized, and the recognition event invokes            a delineating demarcation between the recognized tactile            gesture and the next tactile gesture to follow);    -   Special ending or starting attribute to gestures;    -   Special delimiting or entry-action gesture(s)—for example        lift-off, tap with another finger, etc.

3.2.4 “Intra-Gesture Prosody”

Additionally, because of the temporal aspects of gestures and thegestemes they comprise, aspects of gesture rendering over time can bemodulated as they often are in speech, and thus gestures also admit achance for formal linguistic “prosody” to be imposed on gestures forconveyance of additional levels of meaning or representations of aparameter value. Intra-gesture and Inter-gesture prosody are furthertreated in co-pending U.S. Patent Application 61/567,626.

The HDTP approach to touch-based user interfaces allows for there to beyet other additional content to be imposed in such simple “X” or “+”gestures. For example:

-   -   At least one contact angle (yaw, roll, pitch) of the finger(s)        used to render each of the strokes of the “X” or “+” gesture;    -   How many fingers used to render each of the strokes of the “X”        or “+” gesture;    -   Embellishment in individual component element rendering (angle        of rendering, initiating curve, terminating curve,        intra-rendering curve, rates of rendering aspects, etc.);    -   Variations in the relative location of individual component        element rendering;    -   What part(s) of the finger or hand used to render each of the        strokes of the “X” or “+” gesture;    -   Changes in one or more of the above over time.        A ‘natural’ potential name for at least some of these could be        “intra-gestural prosody.” This term could be extended to include        the entire list, or another term could be forum

3.3 Summarizing Comparative View

The table below comparatively summarizes selected aspects the constructsmade thus far for gestures in relation to the corresponding attributesin established phonetic and orthographic linguistics.

Written Word Gesture Spoken Word Primitive Element grapheme “gesteme”phoneme Delimiters gaps and prosody/lexicon prosody/ punctuation lexiconSerializing Media space time and space time Feedback direct markingdirect marking and/ indirect action or indirect action

3.4 Relations to Earlier Pen-Based Interfaces and HandwritingRecognition

In that gestures involve time-varying touch actions (typically executedwith a finger), it is also appropriate to consider relations betweentouch-based gestures and earlier efforts directed to pen-basedinterfaces and real-time handwriting recognition (typically executedwith a stylus). An early (1961) example of an effort directed tohandwriting recognition is that of Eden [3] which will be consideredlater.

4. Gesture Executions and their Renderings in Measured Signal Space

FIG. 6 depicts a representation of a general user interface arrangementrelevant to the present invention. Physical executions of gestures by auser are observed through a sensing process by a sensor. The sensor andits electrical interface provide raw signals (analog, digital, bytestreams, image frames, etc.) to software elements, likely includingsoftware drivers (not shown) and measurement software algorithms. Themeasurement software algorithms produce measured signals (which can alsobe viewed and referred to as “measurement signals,” “measured values,”and “measurement values”) that are made available for subsequentprocessing and for use by applications. The measured signals (or“measurement signals,” “measured values,” and “measurement values”) arealso presented to calculation software algorithms. The calculationsoftware algorithms produce calculated signals (which can also be viewedand referred to as “calculated values”) that are made available forsubsequent processing and for use by applications.

As taught in U.S. Pat. No. 6,570,078 and U.S. patent application Ser.No. 11/761,978:

-   -   A touch posture can be viewed as a recognized tactile image        pattern (for example as measured by a touch sensor).    -   A touch gesture can be viewed as a time-varying tactile image        pattern with recognized dynamic changes over time (such as a        finger flick, single-finger double-tap, etc.).

It is also noted that U.S. Pat. No. 6,570,078 and U.S. patentapplication Ser. No. 11/761,978 extend these notions from touch sensorsto include gestures rendered as verbal hand signals (for example asmeasured by a video camera) as well as, for example, facial expressionsand lip movements.

In a general view, then:

-   -   From the user experience viewpoint, a gesture is rendered by a        user as a time-varying pattern or trajectory of movement in the        physical space measured or observed by the user interface        sensor.    -   From the machine viewpoint, a gesture as measured by an        associated sensor can be represented as a time-varying pattern        or trajectory in a measured signal space.

For example, a touch-based finger flick, wherein a finger contact startsin a first (initiating) measured location on a touch surface, andsubsequently moves remaining in contact with the touch surface to asecond (terminating) measured location within a predefined minimuminterval of time, creates a corresponding trajectory in measured signalspace.

Further as to this example, FIG. 7a through FIG. 7c depictrepresentations of an example touch-based single-finger “finger-flick”gesture, wherein a finger makes physical contact begins in a first(initiating) location on a touch surface, and moves remaining in contactwith the touch surface to a second (terminating) location roughly alonga straight-line path within a predefined minimum interval of time.

As another example, FIG. 8a through FIG. 8c depict representations of anexample touch-based single-finger hook-shaped gesture, wherein a fingermakes physical contact begins in a first (initiating) location on atouch surface, and moves remaining in contact with the touch surfacealong hook-shaped path to a second (terminating) location within apredefined minimum interval of time.

FIG. 9 depicts an example signal-space representation of thesingle-finger “finger-flick” gesture represented by FIG. 7a through FIG.7c , wherein a signal-space trajectory starts in a first (initiating)signal-space location and changes values to a second (terminating)signal-space location within a predefined minimum interval of time.Similarly, FIG. 10 depicts an example signal-space representation of thesingle-finger hook-shaped gesture represented by FIG. 8a through FIG. 8c, wherein a signal-space trajectory starts in a first (initiating)signal-space location and changes values to a second (terminating)signal-space location within a predefined minimum interval of time.

The concepts of represented in FIG. 7a through FIG. 7c , FIG. 8a throughFIG. 8c , FIG. 9, and FIG. 10, are purely simple representative examplesreadily more generally extended to comprise more dimensions, parameters,and/or other types of measurements or values calculated from measuredvalues for arbitrary sensors, gesture actions, and signal spaces.Accordingly, the invention is hardly limited to the examples representedin FIG. 1a through FIG. 1c , FIG. 8a through FIG. 8c , FIG. 9, and FIG.10. As one example extension, the signal-space can be expanded toinclude rates of change (such as velocity and/or acceleration) ofcalculated from measured values. As another example extension, an HDTPor other high-dimensional gesture user interface arrangement such asthose taught in U.S. Pat. No. 6,570,078, U.S. patent application Ser.Nos. 11/761,978, and 12/418,605, can be used as a user interfaceparadigm. Such arrangements expand and/or alter the number and/or typeof measurements or values calculated from measured values for morearbitrary types of sensors, gesture actions, and signal spaces.

As taught in U.S. patent application Ser. No. 12/418,605, one or moremeasured or calculated values and/or the rate of change over time of oneor more of these measured or calculated values can be individually, incombination, or within a numerical computation, submitted to one or morethreshold tests, wherein the outcomes of the threshold tests can betreated as symbols. Accordingly, in a simple implementation, symbolsthus created by threshold tests that do not comprise threshold tests onrates of change can be viewed as postures, while symbols created bythreshold tests that do comprise threshold tests on rates of change canbe viewed as gestures. In more sophisticated implementation, symbolscreated by threshold tests that comprise threshold tests requiring ratesof change to be higher than a reference value can be viewed as gestures,while symbols created by threshold tests that comprise threshold testsrequiring rates of change to be lower than a (same of different)reference value can be viewed as postures. U.S. patent application Ser.No. 12/418,605 also teaches that the threshold tests can comprise thosewherein the velocity or acceleration of a measured value or calculatedvalue exceeded a specified reference value. Additionally, U.S. patentapplication Ser. No. 12/418,605 also teaches the generation of symbolsby shape recognition functions, and that one or both of threshold testsand shape recognition can be adapted to generate more than one symbol ata time (for example, several conditions may be satisfied at the samemoment).

Alternatively, a symbol is determined by the outcome of a vectorquantizer applied to one or more measured or calculated value(s)responsive to a user interface sensor.

Alternatively, a symbol is determined by the outcome of a matched filterapplied to one or more measured or calculated value(s) responsive to auser interface sensor.

In general, each individual gesture comprises some sort ofgesture-beginning and corresponding gesture-end. For example, in oneembodiment a gesture-beginning can be defined as the event of thebeginning of measured contact with a touch sensor for a contiguousinterval of time and the corresponding gesture-end can be defined as theevent of the ending of measured contact with a touch sensor for thatcontiguous interval of time. As another example, in an embodiment agesture-beginning can be defined as the event of the rate of change ofat least one measured or calculated value exceeding a reference valueand the corresponding gesture-end can be defined as the event of therate of change of at least one measured or calculated value droppingbelow a (same or different) reference value. As yet another example,aspects of the two preceding embodiments can be logically combined, forexample using a logic operation (such as “AND” or “OR”) on measuredcontact events and rate of change events. As yet another example, in anembodiment a gesture-beginning can be defined as the event of generationof a designated symbol and the corresponding gesture-end can be definedas the event of generation of a (same or different) designated symbol.As yet another example, aspects of the last embodiment and first twopreceding embodiments can be logically combined, for example using alogic operation (such as “AND” or “OR”) on two or more measured contactevents, rate of change events, and symbol generation events.

5. Example Signal Spaces, Symbol Generation, and Parameter Generation inSimple and High-Dimensional User Interfaces

FIG. 11 depicts an example symbol generation arrangement for generatinga sequence of symbols from (corrected, refined, raw, adapted,renormalized, etc.) real-time measured parameters values provided byother portions of an HDTP system. Referring to FIG. 11, one or more(here all are shown) of (corrected, refined, raw, adapted, renormalized,etc.) real-time measured values and/or calculated values of HDTPparameters associated with a tactile sensor blob or constellation ofblobs (here these are represented by the set of measured values and/orcalculated values of finger posture parameters {x, y, p, φ, θ, ψ}(thesecorresponding respectively to left-right, forward-back, downwardpressure/displacement, roll angle, pitch angle, and yaw angle of thefinger with respect to the touch sensor surface) are differenced,numerically differentiated, etc. with respect to earlier values so as todetermine the rate of change (shown here per time step although thiscould be per unit time, a specified number of time steps, etc.).

Further details of HDTP output parameters responsive, for example, totouch by the human hand is provided in at least the following co-pendingpatent applications:

-   -   U.S. patent application Ser. No. 12/724,413;    -   U.S. patent application Ser. No. 13/038,372;    -   U.S. patent application Ser. No. 13/180,512;    -   U.S. Patent Application 61/506;    -   U.S. Patent Application 61/567,626;    -   U.S. Patent Application 61/522,239;    -   U.S. patent application Ser. No. 13/093,834.

Alternatively, or in other types of user interface arrangements, agreater or lesser number and/or alternate collection of parameters canbe used).

Both the real-time measured values of HDTP output parameters and one ormore rate of change outputs are provided to a plurality of conditionaltests. In one implementation or mode of operation, none of theseconditions from the plurality of conditional tests overlap. In otherimplementations or modes of operation, at least two of the conditionsfrom the plurality of conditional tests overlap.

Additionally, the invention provides for conditions that are equivalentto the union, intersection, negation, or more complex logical operationson simpler conditional tests. For example, a conditional test comprisingan absolute value of a variable can be implemented as a logicaloperation of simpler conditional test. Note this is equivalent toallowing a symbol to be associated with the outcome of a plurality oftests, also provided for by the invention in more general terms.

In the example implementation depicted in FIG. 11, each time a conditionis met a symbol corresponding to that condition is generated as anoutput. Note that in principle more than one symbol can be generated ata time.

In some implementations (for example, if none of the conditions overlap)at most one symbol can be generated at any given moment. The symbol canbe represented by a parallel or serial digital signal, a parallel orserial analog signal, a number, an ASCII character, a combination ofthese, or other representation. In some implementations the symbol isgenerated when the condition is first met. In other implementations, thesymbol is maintained as a state throughout the time that the conditionis met. Note that it is possible in some implementations for no symbolto be generated (for example in some implementations if no conditionshave been met, or in some implementations if conditional test outcomeshave not changed since an earlier symbol was generated, etc.).

In other implementations, a symbol can be generated only under thecontrol of a clock or sampling command, clock signal, event signal, orother symbol generation command. FIG. 12 depicts a modification of theexemplary arrangement of FIG. 11 wherein symbol can be generated onlyunder the control of a clock or sampling command, clock signal, eventsignal, or other symbol generation command.

In some implementations or modes of operation, some symbols aregenerated by the approach depicted in FIG. 11 while other symbols aregenerated by the approach depicted in FIG. 12. Either of thesearrangements used individually or both arrangements used together are inaccordance with the general exemplary arrangement depicted in FIG. 13,adapted from U.S. patent application Ser. No. 12/418,605.

Further details of HDTP concepts and implementation examples fordelimiters and symbols that are responsive, for example, to touch by thehuman hand is provided in at least the following co-pending patentapplications:

-   -   U.S. patent application Ser. No. 13/180,512;    -   U.S. Patent Application 61/506,096;    -   U.S. Patent Application 61/567,626;    -   U.S. Patent Application 61/522,239;    -   U.S. patent application Ser. No. 13/093,834;    -   U.S. patent application Ser. No. 13/038,365.

It is anticipated that other arrangements for generation of symbols from(corrected, refined, raw, adapted, renormalized, etc.) real-timemeasured parameters values provided by other portions of a userinterface system.

As a very simple yet representative example of symbol generation, assumea particular parameter or rate value, denoted here as “q” is tested (aspart of a more complex conditional tests, as stand alone conditionaltests, etc.) is tested for three conditions:

CASE 1: q<Q_(a)

CASE 2: Q_(a)<q<Q_(b)

CASE 3: q>Q_(b)

FIG. 14 depicts such a conditional test for a single parameter or ratevalue q in terms of a mathematical graph, separating the full range of qinto three distinct regions. The region divisions are denoted by theshort dashed lines. For the sake of illustration Q_(a) could be anegative value and Q_(b) could be a positive value, although this doesnot need to be the case.

Next, consider example sets of conditional test for two values, eitherone of which can be a parameter value or rate value. As a simpleexample, each of the two values can be tested for three conditions in asimilar fashion as for the single value example considered above. FIG.15a depicts such a conditional test for a two values (parameter and/orrate) in terms of a mathematical graph, separating the full range ofeach of the two values into three regions. The region divisions each ofthe two values are denoted by the short dashed lines, for the sake ofillustration one in a negative range for the value and the other in apositive value, although this does not need to be the case. By extendingthe short dashed lines to longer lengths as shown in FIG. 15b , it canbe seen that the region (here a portion of a plane) defined by the fullrange of the two values is divided into 3×3=9 distinct regions.

Similarly, consider example sets of conditional test for three values,any one of which can be a parameter value or rate value. As a simpleexample, each of the three values can be tested for three conditions ina similar fashion as for the examples considered above. FIG. 16a depictssuch a conditional test for a two values (parameter and/or rate) interms of a mathematical graph, separating the full range of each of thethree values into three regions. The region divisions each of the threevalues are denoted by the short dashed lines, for the sake ofillustration one in a negative range for the value and the other in apositive value, although this does not need to be the case. By extendingthe short dashed lines to longer lengths as shown in FIG. 16b , it canbe seen that the region (here a portion of 3-space) defined by the fullrange of the three values is divided into 3×3×3=27 distinct regions.

In a similar way, if there are N variables, each of which are tested forlying within M distinct ranges, the number of distinct regions is givenby M^(N). Thus for six (N=6) parameters (such as for example the six {x,y, p, φ, θ, ψ} provided for each “blob” in a HDTP system), each of whichare tested for lying within distinct ranges (M=3) such as “mid range”and two opposite “far extremes”, the number of distinct regions is givenby 3⁶=729.

In principle, each the corresponding rate (time-derivative) values foreach of the parameters could be split into three ranges as well. Apractical distinction among rates from a user's viewpoint might beseparate recognition of a “zero or slow” and “anything fast” rate (M=2).Such a conditional test could utilize an absolute value function in theconditional test. Note that a two-value test on an absolute value isequivalent to a three range test wherein the two extreme ranges producethe same outcome. Note the number of distinct regions for the set of sixrate values (N=6), each separately tested for occupancy in two ranges(“zero or slow” and “anything fast”, so M=2) is 2⁶=64.

For an example implementation combining these two aforedescribedexamples, the total number of distinction recognizable regions is729×64=46,656. In principal a distinct symbol could be assigned to eachof these regions, noting that each region is equivalent to a 12-variable(six parameter values plus rate-of-change value for each, giving 12)conditional test outcome. This provides a very rich environment fromwhich to draw for design choices of ergonomics, metaphors, omittedconditions/regions that are not useful or applicable, imposed contextualinterpretations, etc.

It is to be understood that the above is merely a chain of examples andnot to be in any way considered limiting.

5.1 Discrete (Symbol) and Continuous Parameters (Adapted from Ser. No.12/418,605)

The HDTP provides for the production of the following six parametervalues from a single blob associated with the hand or other pliableobject:

-   -   Calculation of downward pressure and two planar centers of        contact area;    -   Calculation of roll, pitch, and yaw angles of contact area.

In some embodiments, these parameter values may take on a wider range(i.e., more than 3 and typically far greater than 2) of numerical valueswithin a consecutive range—in that they are range of numerical valuespossible, the values taken on by these six parameters will be informallyreferred to as “continuous” (in contrast to a smaller set of binaryvalues, or a set of non-consecutive “symbols”).

These parameter values may be numerically differentiated in time (forexample, by simply taking the difference between values of the currentand previous scan) to produce rate measurements for the parameters, suchas velocity and (by numerically differentiating velocity) acceleration.These result in additional “continuous” rate values.

One or more parameter values and/or rate values may be individually, incombination, or within a numerical computation, submitted to one or morethreshold tests. The outcomes of the threshold tests may be regarded assymbols (for example, what region of the sensor array is the center ofcontact in, has a roll angle velocity or acceleration exceeded aspecified value, etc.).

Additionally, aforementioned shape recognition functions may alsogenerate symbols. The invention provides for one or both of thethreshold and shape recognition elements to generate more than onesymbol at a time (for example, several conditions may be satisfied atthe same moment).

5.2 Delimiters, Sampling (Adapted from Ser. No. 12/418,605)

The invention affords and provides for yet further capabilities. Forexample, FIG. 17 shows an intrinsic metaphor applied to a touch sensorthat senses touch attributes, and these being directed to an imposedmetaphor causing an application response to be invoked on an associatedapplication.

As another example, a sequence of symbols can be directed to a statemachine, as shown in FIG. 18, to produce other symbols that serve asinterpretations of one or more possible symbol sequences. In anembodiment, one or more symbols may be designated the meaning of an“Enter” key, permitting for sampling one or more varying parameter,rate, and/or symbol values and holding the value(s) until, for example,another “Enter” event, thus producing sustained values as illustrated inFIG. 19.

In an embodiment, the symbols produced by arrangements such as that ofFIG. 13 include symbols that are responsive to rate values. In someembodiments, these rate-responsive symbols can directly serve asrecognitions or signifiers of simple gestures, for example a “fingerflick” in a suitably limited gesture lexicon.

5.3 Support for Discrete Grammars (Adapted from Ser. No. 12/418,605)

FIG. 13, introduced earlier, illustrates an exemplary embodiment ofthese approaches. This demonstrates that simple contact with (or otheroperative stimulus of) the sensor array can produce a rich informationflux of parameter, rate, and symbol values.

Together with the rich metaphors available with the touch interface, atremendous range of synergistic user interface opportunities areprovided by the present invention. Further processing opportunitiessupporting a full range of postures, gestures, real-time parameterextractions, and information needed for implementations of gesturegrammars is depicted within a portion of FIG. 20.

5.4 Support for Continuous-Grammar (Adapted from Ser. No. 12/418,605)

As an additional syntactic aspect, specific hand postures and/orgestures may mapped to specific selected assignments of control signalsin ways affiliated with specific purposes. For example, finger ends maybe used for one collection of . . . parameters, thumb for a secondpotentially partially overlapping collection of . . . parameters, flatfingers for a third partially-overlapping collection, wrist for afourth, and cusp for a fifth, and fist for a sixth. In this case it maybe natural to move the hand through certain connected sequences ofmotions; for example: little finger end, still in contact, dropping toflat-finger contact, then dropping to either palm directly or first tocusp and then to palm, then moving to wrist, all never breaking contactwith the touch-pad. Such permissible sequences of postures that can beexecuted sequentially without breaking contact with the touch-pad willbe termed “continuous grammars.”

To support the handling of continuous grammars, it is useful to set upparameter assignments, and potentially associated context-sensitiveparameter renormalizations, that work in the context of selected (or allavailable) continuous grammars. For example, as the hand contact evolvesas being recognized as one posture and then another, parameters may besmoothly handed-over in interpretation from one posture to anotherwithout abrupt changes, while abandoned parameters either hold theirlast value to return to a default value (instantly or via a controlledtransient.

5.5 Context (Adapted from Ser. No. 12/418,605)

In an embodiment, one or more symbols may be designated as setting acontext for interpretation or operation and thus control mapping and/orassignment operations on parameter, rate, and/or symbol values as shownin FIG. 21.

The operations associated with FIG. 18, FIG. 19, and FIG. 21 can becombined to provide yet other capabilities. For example, the exemplaryarrangement of FIG. 22 shows mapping and/or assignment operations thatfeed an interpretation state machine which in turn controls mappingand/or assignment operations. In implementations where context isinvolved, such as in arrangements such as those depicted in FIG. 18,FIG. 19, FIG. 21 and FIG. 22, the invention provides for bothcontext-oriented and context-free production of parameter, rate, andsymbol values. The parallel production of context-oriented andcontext-free values may be useful to drive multiple applicationssimultaneously, for data recording, diagnostics, user feedback, and awide range of other uses.

5.6 Gesture Compositions and Deconstructions with Respect to PrimitiveElements in Measured Signal Space

Among the gesture linguistic concepts taught U.S. patent applicationSer. No. 12/418,605 is that a sequence of symbols can be directed to astate machine to produce other symbols that serve as interpretations ofone or more possible symbol sequences. This provides one embodiment ofan approach wherein (higher-level) gestures are constructed fromprimitive elements, in this case, other (lower-level) gestures. In suchan arrangement, a predefined gesture can comprise a specific sequence ofplurality of other gestures. For example FIG. 23 depicts an examplerepresentation of a predefined gesture comprised by a specific sequenceof three other gestures. Similarly, a predefined gesture comprised by aspecific sequence of two other gestures, or a predefined gesturecomprised by a specific sequence of four or more other gestures.

In an embodiment, a specific predefined gesture is comprised by aparticular predefined sequence of gestemes. FIG. 24 depicts an examplerepresentation of a predefined gesture comprised by a sequence of fiverecognized gestemes. Similarly, a predefined gesture comprised by aspecific sequence of two, three, or four gestemes, or a predefinedgesture comprised by a specific sequence of six or more other gestemes.Additionally, in some arrangements a predefined gesture can be comprisedby a single gesteme.

In an embodiment, a recognized gesteme is comprised of a symbol producedby one or more threshold test(s) applied to one or more measured orcalculated value(s) responsive to a user interface sensor.

In an embodiment, a recognized gesteme is comprised of a sequence ofsymbols produced by one or more threshold test(s) applied to one or moremeasured or calculated value(s) responsive to a user interface sensor.

In an embodiment, a recognized gesteme is comprised of a symbol producedby a state machine, the state machine responsive to a sequence ofsymbols produced by one or more threshold test(s) applied to one or moremeasured or calculated value(s) responsive to a user interface sensor.

In an embodiment, a recognized gesteme is determined by the outcome of avector quantizer applied to one or more measured or calculated value(s)responsive to a user interface sensor.

In an embodiment, a recognized gesteme is determined by the outcome of amatched filter applied to one or more measured or calculated value(s)responsive to a user interface sensor.

5.7 Example HDTP Parameters that can be Generated byErgonomically-Viable Single-Hand Compound Postures (Adapted from Ser.No. 12/418,605)

There are many ways to organize the possible degrees of freedomgenerated by ergonomically-viable single-hand compound postures. Oneexemplary organization is to first consider the overall orientationattributes of the entire compound posture, and then consider thefinger-by-finger variations that are possible with respect to it. Thisapproach has several variations, a few of which are presented here.

The overall orientation attributes of the entire compound posture mayinclude one or more of the following:

-   -   Overall Positions/Displacements of the Compound Posture:        -   left-right position or translation;        -   forward-back position or translation;        -   more-less downward displacement or translation (pressure);    -   Overall Angles/Rotations of the Compound Posture:        -   pivoting rotation (yaw);        -   left-right tilt (roll);        -   forward-back tilt (pitch).

These overall compound posture parameters may be obtained by variousmeans, some of which as discussed above. These include selectingparameters individually calculated for a representative finger ornon-finger region, averaging individually calculated parameters, and/ormerging at least some running sums at the data acquisition stage.

The finger-by-finger differential variations that are possible withrespect to the overall orientation attributes of an entire compoundposture (including ones that involve most or all of the fingers lyingflat) may include one or more of the following:

-   -   separation angle of adjacent fingers;    -   difference in downward pressure.

This approach gives up to two extra parameters for each added finger. Ina more sophisticated approach for arched finger postures, thefinger-by-finger differential variations that are possible with respectto the overall orientation attributes of the entire compound posture mayinclude one or more of the following:

-   -   difference in left-right position;    -   difference in forward-back position;    -   difference in downward pressure

This approach gives up to three extra parameters for each added finger.Thus, most generally, for a single-hand compound posture employing N ofthe five fingers of the hand, the maximum number of independentparameters that can be independently controlled at the same time is inthe range of 6+2(N−1) to 6+3(N−1). For five fingers, this gives amaximum of fourteen parameters to as many as eighteen parameters for anarched single-hand posture. The number of parameters can be yet furtherexpanded by including the palm and the wrist.

The invention provides for the expansion of the single blob version ofFIG. 13 so as to provide parameter calculations for the cases ofmultiple independent individual blobs and/or compound image blobs. Thetop portion of FIG. 13 depicts an example embodiment wherein sensor datacan be interpreted as one blob, two or more individual blobs, or as acompound posture. These may be calculated in parallel and/orselectively, and in selective modes the selection may be controlled bythe application using a control arrangement like that of FIG. 21 or bysemantic feedback using a control arrangement similar to FIG. 22.

5.8 Layered and Multiple-Channel Posture-Level Metaphors

The invention provides for various types of layered and multiple-channelmetaphors. Layered metaphors at higher semantic and grammatical levelswill be considered later. FIG. 25 depicts a representation of a layeredand multiple-channel metaphor wherein the {x,y} location coordinatesrepresent the location of a first point in a first geometric plane, andthe {roll,pitch} angle coordinates are viewed as determining a secondindependently adjusted point on a second geometric plane. In variousversions of such metaphors, one or more of the following can beincluded:

-   -   the first and second planes can be viewed as being superimposed        (or alternatively, entirely independent)    -   The yaw angle can be viewed as affecting the angle of rotation        of one plane with respect to another (or alternatively, entirely        independent)    -   The pressure exerted or associated displacement can be viewed as        affecting the separation distance between the planes (or        alternatively, entirely independent).

5.9 Compound Parameter, Rate, and Symbol Production (Adapted from Ser.No. 12/418,605)

The invention provides for the expansion of the single blob version ofFIG. 13 so as to provide shape and posture recognition calculations forthe cases of multiple independent individual blobs and/or compound imageblobs. The bottom portion of FIG. 13 depicts an example embodimentwherein sensor data can be interpreted as one blob, two or moreindividual blobs, or as a compound posture. These may be calculated inparallel and/or selectively, and in selective modes the selection may becontrolled by the application using a control arrangement like that ofFIG. 21 or by semantic feedback using a control arrangement like that ofFIG. 22.

6. Support for Affixes and Interrupted Gestures

In spoken and written language, various types of affixes are commonlyused. Some of the types of affixes found in linguistic theory of wordsinclude:

-   -   Suffix: addendum appended to the end of a root word;    -   Prefix: addendum appended to the beginning of a root word;    -   Infix: addendum inserted within a root word;    -   Circumfix: first addendum appended to the beginning of a root        word and second associated addendum appended to the end of a        root word;    -   Interfix: addendum inserted between two root words;    -   Transfix: An affix that incorporates a pause delineating between        a root word and addendum or insertions.

The present invention provides support for the structuring, recognition,processing, and interpretation of “gesture affixes”, such as:

-   -   Gesture suffix: addendum appended to the end of a gesture;    -   Gesture prefix: addendum appended to the gesture word;    -   Gesture circumfix: first addendum appended to the beginning of a        first gesture and second associated addendum appended to the end        of a second gesture;    -   Gesture interfix: addendum inserted between two gestures;    -   Gesture transfix: An affix that incorporates a pause delineating        between a gesture and addendum or insertions.

Various implementation approaches can be used, and a number of examplesare provided. As one type of approach, one or more gestemes can be usedto create the addendums. In various implementations, the addendums canbe gestemes that are not recognized as gestures, sequences of gestemesthat are not recognized as gestures, gestemes that are not recognized asgestures, sequences of gestemes that are not recognized as gestures,and/or combinations of these.

It is noted that some gesture affixes, such as gesture infixes, involveinterruption of the execution of a gesture, while other gesture affixes,such as gesture suffixes and gesture prefixes, do not involveinterruption of the execution of a gesture. There are also other reasonsfor supporting the interruption of the execution of a gesture that haveno relation to gesture affixes. FIG. 26 depicts a representation of therelations between gesture affixes and interrupted gesture executions.Interrupted gestures can also be more broadly supported by the presentinvention so as address covering non-affix cases.

6.1 Gesture Suffixes

In the gesteme implementation of gestures, a first gesture G^(A)comprises a first sequence of m gestemes {g₁ ^(A), . . . , g_(m) ^(A)}.Upon completion of the execution of the first gesture by the user, asecond gesteme g₁ ^(B) or sequence of n gestemes {g₁ ^(B), . . . , g_(n)^(B)} will be executed. Upon the completion of the execution of thesecond gesteme or sequence of gestemes, the execution of the remainingunexecuted gesteme(s), the first gesture will be recognized as having agesture suffix.

In some implementations, only a single gesteme is permitted as a suffix.In other implementations, only a specific gesteme or sequence ofgestemes is permitted as a suffix. In yet other implementations, only aspecific gesteme or sequence of gestemes is permitted as a suffix. Inyet other implementations, a wider range of gestemes or sequence ofgestemes is/are permitted as a suffix.

More explicitly, this includes the following cases for the compositesequence of gestemes:

-   -   {g₁ ^(A), g₁ ^(B)} where m=1 and n=1;    -   {g₁ ^(A), g₁ ^(B), . . . , g_(n) ^(B)} where m=1 and n>1;    -   {g₁ ^(A), . . . , g_(m) ^(A), g₁ ^(B)} wherein m>1 and n=1;    -   {g₁ ^(A), . . . , g_(m) ^(A), g₁ ^(B), . . . , g_(n) ^(B)} where        m>1,and n>1;        Note that cases with m=1 are those wherein the first gesture        comprises exactly one gesteme, and cases with n=1 are those        wherein the suffix comprises exactly one gesteme.

In an embodiment, the existence of any of the above-listed interruptioncases is used to convey semantic content.

In an embodiment, the point of the interruption within the trajectory ofthe first gesture is used to convey semantic content.

In an embodiment, the point of the interruption within the gestemesequence of the first gesture is used to convey semantic content.

Alternatively, semantic pattern recognition or other approaches can beused.

6.2 Gesture Prefixes

In the gesteme implementation of gestures, a first gesture G^(A)comprises a first sequence of m gestemes {g₁ ^(A), . . . , g_(m) ^(A)}.

Prior to the execution of the first gesture by the user, a secondgesteme g₁ ^(B) or sequence of n gestemes {g₁ ^(B), . . . , g_(n) ^(B)}will be executed. Upon the completion of the execution of the firstgesture, the first gesture will be recognized as having a gestureprefix.

In some implementations, only a single gesteme is permitted as a prefix.In other implementations, only a specific gesteme or sequence ofgestemes is permitted as a prefix. In yet other implementations, only aspecific gesteme or sequence of gestemes is permitted as a prefix. Inyet other implementations, a wider range of gestemes or sequence ofgestemes is/are permitted as a gesture prefix.

More explicitly, this includes the following cases for the compositesequence of gestemes:

-   -   {g₁ ^(B), g₁ ^(A)} where m=1 and n=1;    -   {g₁ ^(B), g₁ ^(A), . . . , g_(m) ^(A)} where m>1 and n=1;    -   {g₁ ^(B), . . . , g_(n) ^(B), g₁ ^(A)} wherein m=1 and n>1;    -   {g₁ ^(B), . . . g_(n) ^(B), g₁ ^(A), . . . , g_(m) ^(A)} where        m>1,and n>1;        Note that cases with m=1 are those wherein the first gesture        comprises exactly one gesteme, and cases with n=1 are those        wherein the prefix comprises exactly one gesteme.

In an embodiment, the existence of any of the above-listed interruptioncases is used to convey semantic content.

In an embodiment, the point of the interruption within the trajectory ofthe first gesture is used to convey semantic content.

In an embodiment, the point of the interruption within the gestemesequence of the first gesture is used to convey semantic content.

Alternatively, semantic pattern recognition or other approaches can beused.

6.3 Gesture Affixes realized via Interruption of the Execution of aFirst Predefined Gesture with the Execution of At Least One Gesteme

The execution of a gesture can be interrupted by the user executing it.The executed gesture can be resumed or not resumed (i.e., abandoned).

In an embodiment, a partially-executed gesture can be recognized as suchand information regarding the partially-executed gesture (as measuredand/or subsequently-interpreted) is stored. In an embodiment, should thepartially-executed gesture be resumed, the stored information regardingthe partially-executed gesture is used in the recognition of thecompleted form of the previously partially-executed gesture. In anembodiment, should the partially-executed gesture not be resumed, thestored information regarding the partially-executed gesture is deleted.In an embodiment, should the partially-executed gesture not be resumedwithin a pre-determined length of time, the stored information regardingthe partially-executed gesture is deleted.

With this established, this section considers the case wherein theexecution of a first predefined gesture is interrupted, one or moregestemes that is/are not collectively recognized as gesture are thenexecuted, and the execution of the first predefined gesture is resumedand completed. If the additional gesteme(s) thus “inserted” during theinterruption is/are used linguistically as an affix to the firstgesture, the latter case amounts to the “infix” case of an “affix” inthe context of gesture grammars.

The case wherein the execution of a first predefined gesture isinterrupted, one or more second gesture(s) is/are then fully executed,and the execution of the first gesture is resumed and completed isaddressed in co-pending U.S. patent application Ser. No. 13/414,600.Those teachings are selectively used below as cited.

In an aspect of the invention, the additional gesteme(s) thus “inserted”during the interruption is/are used in the context of a gesture grammaras an affix to the first gesture as an “infix.”

As an example, FIG. 27a through FIG. 27j depict an examplerepresentation of the execution of a first example predefined gesturethat is begun (FIG. 27a ) and interrupted (FIG. 27b and FIG. 27c ), thefull execution of an example second predefined gesture (FIG. 27d , FIG.27e , FIG. 27f , and FIG. 27g ), and the resumed and completed executionof the first predefined gesture (FIG. 27h , FIG. 27i , and FIG. 27j ).

In this example as described thus far, recognition of the interruptionof the execution the first gesture is established at least by thelift-off of the finger from the touch surface depicted in FIG. 27c .Recognition of the interruption of the execution the first gesture canalso or alternatively require semantic restrictions on gesteme sequencesfor predefined gestures. In either version (pause version or semanticrestriction version) of this variation where the FIG. 27c lift-off isskipped, the second gesture must begin at the same location where thefirst gesture was interrupted. Note by including the FIG. 27c lift-off,the location of the beginning of the second gesture need not beinfluenced by the location where the first gesture was interrupted.

Similarly, in an embodiment the lift-off of the finger from the touchsurface depicted in FIG. 27g can be used to signify the completion ofthe second gesture and the prompt for the completion of the firstgesture. In a variation of this example, the lift-off of the finger fromthe touch surface depicted in FIG. 27g can be omitted; instead a pausecan be used to signify the completion of the second gesture and theprompt for the completion of the first gesture, or semantic restrictionson gesteme sequences can be used to signify the completion of the secondgesture and the prompt for the completion of the first gesture. Ineither version (pause version or semantic restriction version) of thisvariation where the FIG. 27g lift-off is skipped, the resumption of thefirst interrupted gesture must begin at the same location where thesecond gesture ended. Note by including the FIG. 27g lift-off, thelocation of the resumption of the first interrupted gesture need not beinfluenced by the location where the second gesture was completed.

Similarly, in an embodiment the lift-off of the finger from the touchsurface depicted in FIG. 27j can be used to signify the completion ofthe first gesture. In a variation of this example, the lift-off of thefinger from the touch surface depicted in FIG. 27j can be omitted;instead a pause can be used to signify the completion of the firstgesture, or semantic restrictions on gesteme sequences can be used tosignify the completion of the first gesture.

As a second example, FIG. 28a through FIG. 28j depict a variation on theexample of FIG. 27a through FIG. 27j wherein the lift-off eventsdepicted by FIG. 27c , FIG. 27g , and FIG. 27j are replaced with thepause events depicted in FIG. 28c with FIG. 28d , FIG. 28g with FIG. 28h, and in FIG. 28j . Such pause events can be recognized by conditionswherein the magnitude of the rate-of-change of one or more measuredvalues or the magnitude of the rate-of-change of one or more valuescalculated from one or more measured values fall below associatedreference threshold(s). In another variation of this example, thelift-off of the finger from the touch surface depicted in FIG. 28j isnot used; instead semantic restrictions on gesteme sequences can be usedto signify the completion of the second gesture and the prompt for thecompletion of the first gesture. FIG. 28a is a repeat of FIG. 27a . FIG.28b is a repeat of FIG. 27b . FIG. 28e is a repeat of FIG. 27e . FIG.28f is a repeat of FIG. 27f . FIG. 28i is a repeat of FIG. 27 i.

As a third example, FIG. 29a through FIG. 29f depict a variation on theexample of FIG. 27a through FIG. 27j wherein the lift-off eventsassociated FIG. 27c , FIG. 27g , and FIG. 27j are omitted altogether andsemantic restrictions on gesteme sequences can be used to signify thecompletion of the second gesture and the prompt for the completion ofthe first gesture. For this example, the second gesture must begin atthe same location where the first gesture was interrupted, and theresumption of the first interrupted gesture must begin at the samelocation where the second gesture ended.

In an embodiment, a method is provided for a user interface recognizingthe interruption of the execution of a first gesture with the executionof a second gesture, the method comprising:

-   -   Receiving measured information from a user interface sensor, the        measured information responsive to user interaction actions made        by a user;    -   Applying at least one operation to the measured information to        produce a sequence of symbols, each symbol produced by the at        least one operation responsive to an associated portion of a        user interaction actions made by the user;    -   Determining from the sequence of symbols that the user's        execution a first gesture has been interrupted;    -   Determining from the sequence of symbols that the user's        execution a second gesture has been started and completed before        the first gesture has been resumed; and    -   Determining from the sequence of symbols that the user's        execution a first gesture has been completed;

wherein the first gesture is recognized and the second gesture isrecognized.

As a second example, in the gesteme implementation of gestures, a firstgesture G^(A) comprises a first sequence of m gestemes {g₁ ^(A), . . . ,g_(m) ^(A)}. This gesture will, at some point in its execution by theuser, be interrupted and a second gesteme g₁ ^(B) or sequence of ngestemes {g₁ ^(B), . . . , g_(n) ^(B)} will be executed. Upon thecompletion of the execution of the second gesteme or sequence ofgestemes, the execution of the remaining unexecuted gesteme(s) of thefirst gesture is resumed and the execution of the first gesture is thencompleted. In various implementations, the second gesteme or sequence ofgestemes can serve as gesture “infix.”

More explicitly, this includes the following cases for the compositesequence of gestemes:

-   -   {g₁ ^(A), g₁ ^(B), . . . , g_(n) ^(B), g₂ ^(A), g₂ ^(A)} where        m=2 and n>1;    -   {g₁ ^(A), g₁ ^(B), . . . , g_(n) ^(B), g₂ ^(B), . . . , g_(m)        ^(A)} where m>2 and n>1;    -   {g₁ ^(A), . . . , g_(m−1) ^(A), g₁ ^(B), . . . , g_(n) ^(B),        g_(m) ^(A)} wherein m>2 and n>1;    -   {g₁ ^(A), . . . , g_(k) ^(A), g₁ ^(B), . . . , g_(n) ^(B),        g_(k+1) ^(A), . . . , g_(m) ^(A)} where m>3, 1<k<(m−1), and n>1;    -   {g₁ ^(A), g₁ ^(B), g₂ ^(A), g₂ ^(A)} where m=2 and n=1;    -   {g₁ ^(A), g₁ ^(B), g₂ ^(A), . . . , g_(m) ^(A)} where m>2 and        n=1;    -   {g₁ ^(A), . . . , g_(m−1) ^(A), g₁ ^(B), g_(m) ^(A)} where m>2        and n=1;    -   {g₁ ^(A), . . . , g_(k) ^(A), g₁ ^(B), g_(k+1) ^(A), . . . ,        g_(m) ^(A)} where m>3, 1<k<(m−1), and n=1.        Note that cases with n=1 are those wherein the interruption        comprises exactly one gesteme. Also note that cases with m=1 are        not admissible since the first gesture must be interrupted and        resumed, thus requiring the first gesture to comprise a minimum        of two gestemes for the first predefined gesture.

Alternatively, semantic pattern recognition or other approaches can beused.

Additionally, it is noted that by reversing the roles of the first andsecond gestures, the resulting modified arrangement can be used tosupport gesture circumfixes.

In another embodiment, the existence of any of the above-listedinterruption cases is used to convey semantic content.

In an embodiment, the point of the interruption within the trajectory ofthe first gesture is used to convey semantic content.

In an embodiment, the point of the interruption within the gestemesequence of the first gesture is used to convey semantic content.

As mentioned above, case wherein the execution of a first predefinedgesture is interrupted, one or more second gesture(s) is/are then fullyexecuted, and the execution of the first gesture is resumed andcompleted is addressed in co-pending U.S. patent application Ser. No.13/414,600. In the material below, this insertion of a second gesturewithin the execution of a first gesture gestures case is considered inthe context of affixes, and in particular (although not restricted to)infixes.

Additionally, it is noted that by reversing the roles of the first andsecond gestures, the resulting modified arrangement can be used tosupport gesture circumfixes.

In the gesteme implementation of gestures, a first gesture G^(A)comprises a first sequence of m gestemes {g₁ ^(A), . . . , g_(m) ^(A)}and a second gesture G^(B) comprises a second sequence of n gestemes {g₁^(B), . . . , g_(n) ^(B)}. More explicitly, as with above, thisarrangement also includes the following cases for the composite sequenceof gestemes:

-   -   {g₁ ^(A), g₁ ^(B), . . . , g_(n) ^(B), g₂ ^(A), g₂ ^(A)} where        m=2 and n>1;    -   {g₁ ^(A), g₁ ^(B), . . . , g_(n) ^(B), g₂ ^(A), . . . , g_(m)        ^(A)} where m>2 and n>1;    -   {g₁ ^(A), . . . , g_(m−1) ^(A), g₁ ^(B), . . . , g_(n) ^(B),        g_(m) ^(A)} wherein m>2 and n>1;    -   {g₁ ^(A), . . . , g_(k) ^(A), g₁ ^(B), . . . , g_(n) ^(B),        g_(k+1) ^(A), . . . , g_(m) ^(A)} where m>3, 1<k<(m−1), and n>1;    -   {g₁ ^(A), g₁ ^(B), g₂ ^(A), g₂ ^(A)} where m=2 and n=1;    -   {g₁ ^(A), g₁ ^(B), g₂ ^(A), . . . , g_(m) ^(A)} where m>2 and        n=1;    -   {g₁ ^(A), . . . , g_(m−1) ^(A), g₁ ^(B), g_(m) ^(A)} where m>2        and n=1;    -   {g₁ ^(A), . . . , g_(k) ^(A), g₁ ^(B), g_(k+1) ^(A), . . . ,        g_(m) ^(A)} where m>3, 1<k<(m−1), and n=1.        Note that cases with n=1 are those wherein the second gesture        comprises only one gesteme. Also note that cases with m=1 are        not admissible since the first gesture must be interrupted and        resumed, thus requiring the first gesture to comprise a minimum        of two gestemes for the first predefined gesture.

Alternatively, semantic pattern recognition or other approaches can beused.

In an embodiment, both the first gesture and second gesture arerecognized.

In an embodiment, the combination of the first gesture and the secondgesture is used to convey additional semantic content beyond that of thefirst gesture and the second gesture in isolation.

In an embodiment, a method is provided for a user interface recognizingthe interruption of the execution of a first gesture with the executionof a second gesture, the method comprising:

-   -   Receiving measured information from a user interface sensor, the        measured information responsive to user interaction actions made        by a user;    -   Applying at least one operation to the measured information to        produce a sequence of gestemes, each gesteme produced by the at        least one operation responsive to an associated portion of a        user interaction actions made by the user;    -   Determining from the sequence of gestemes that the user's        execution a first gesture has been interrupted;    -   Determining from the sequence of gestemes that the user's        execution a second gesture has been started and completed before        the first gesture has been resumed; and    -   Determining from the sequence of gestemes that the user's        execution a first gesture has been completed;        wherein the first gesture is recognized and the second gesture        is recognized.

Additionally, the above aspects of the present invention can be extendedto variation of the above wherein the execution of a first predefinedgesture is interrupted, a sequence of a plurality of other predefinedgestures are then fully executed, and the execution of the firstpredefined gesture is then resumed and completed. More explicitly, inthe gesteme implementation of gestures, this includes the followingcases for the composite sequence of gestemes:

-   -   {g₁ ^(A), g₁ ^(sequence), . . . , g_(n) ^(sequence), g₂ ^(A), g₂        ^(A)} where m=2 and n>1;    -   {g₁ ^(A), g₁ ^(sequence), . . . , g_(n) ^(sequence), g₂ ^(A), .        . . , g_(m) ^(A)} where m>2 and n>1;    -   {g₁ ^(A), . . . , g_(m−1) ^(A), g₁ ^(sequence), . . . , g_(n)        ^(sequence), g_(m) ^(A)} wherein m>2 and n>1;    -   {g₁ ^(A), . . . , g_(k) ^(A), g₁ ^(sequence), . . . , g_(n)        ^(sequence), g_(k+1) ^(A), . . . , g_(m) ^(A)} where m>3,        1<k<(m−1), and n>1;        Here, the first gesture G^(A) comprises a first sequence of m        gestemes {g₁ ^(A), . . . , g_(m) ^(A)} and a sequence of a        plurality of other predefined gestures G^(sequence) comprises a        second sequence of n gestemes {g₁ ^(sequence), . . . , g_(n)        ^(sequence)}, this second sequence being the concatenation of        the gesteme sequences for each gesture in the sequence of other        predefined gestures.

Alternatively, semantic pattern recognition or other approaches can beused.

In an embodiment, all of the first gesture and the sequence of otherpredefined gestures are individually recognized.

In an embodiment, the existence of any of the above-listed interruptioncases is not used to convey semantic content.

The invention provides for various additional operations to be providedbased on any gesture recognitions, the existence of an interruption inthe execution of the first gesture wherein the sequence of otherpredefined gestures is completed during the interruption, details of theinterruption, etc.

In another embodiment, the existence of any of the above-listedinterruption cases is used to convey semantic content.

In an embodiment, the point of the interruption within the trajectory ofthe first gesture is not used to convey semantic content.

In an embodiment, the point of the interruption within the trajectory ofthe first gesture is used to convey semantic content.

In an embodiment, the point of the interruption within the gestemesequence of the first gesture is not used to convey semantic content.

In an embodiment, the point of the interruption within the gestemesequence of the first gesture is used to convey semantic content.

In an embodiment, the combination of the first gesture and the sequenceof other predefined gestures is not used to convey additional semanticcontent beyond that of the first gesture and the second gesture inisolation.

In an embodiment, the combination of the first gesture and the sequenceof other predefined gestures is used to convey additional semanticcontent beyond that of the first gesture and the second gesture inisolation.

In an embodiment, the combination of the first gesture, the sequence ofother predefined gestures, and the location of the interruption withinthe first gesture is used to convey additional semantic content beyondthat of the first gesture and the second gesture in isolation.

6.4 Gesture Circumfixes

As defined at the beginning of the section, a gesture circumfix involvesa first addendum appended to the beginning of a gesture and secondassociated addendum appended to the end of the gesture.

In one type of approach, as noted above, by reversing the roles of thefirst and second gestures in arrangements for the structuring,recognizing, and processing of gesture interfixes, the resultingmodified arrangement can be used to support gesture circumfixes.

In one type of approach, arrangements described above for thestructuring, recognizing, and processing of gesture prefixes and for thestructuring, recognizing, and processing of gesture suffixes can becombined to create an arrangement to support gesture circumfixes

Alternatively, the methods described above for gesture suffixes, gestureprefixes, gesture infixes, and/or gesture circumfixes can be readilymodified and/or combined so as to structure, recognize, and processgesture circumfixes.

Alternatively, semantic pattern recognition or other approaches can beused. 6.5 Gesture Interfixes

As defined at the beginning of the section, a gesture interfix involvesan addendum inserted between two gestures.

In an aspect of the invention, the additional gesteme(s) thus “inserted”during the interruption is/are used in the context of a gesture grammaras an affix to the first gesture as a gesture interfix.

The methods described above for gesture suffixes, gesture prefixes,gesture infixes, and/or gesture circumfixes can be readily modifiedand/or combined so as to structure, recognize, and process gestureinterfixes.

Alternatively, semantic pattern recognition or other approaches can beused.

6.6 Gesture Transfixes

As defined at the beginning of the section, a gesture transfix is anaffix that incorporates a pause delineating between a gesture andaddendum or insertions.

In an aspect of the invention, the additional gesteme(s) thus “inserted”during the interruption is/are used in the context of a gesture grammaras an affix to the first gesture as a gesture transfix.

The methods described above for gesture suffixes, gesture prefixes,gesture infixes, and/or gesture circumfixes can be readily modifiedand/or combined so as to structure, recognize, and process gesturetransfixes.

Alternatively, semantic pattern recognition or other approaches can beused.

7. Fundamentals of Meaning: Morphemes, Lexemes, and Morphology

In traditional linguistics a morpheme is the smallest linguistic unitthat has (semantic) meaning. A word or other next-higher-scalelinguistic unit may be composed of one or more morphemes compose a word.Two basic categories of morphemes relevant to this project are:

-   -   A free morpheme which can function by itself;    -   A bound morpheme which can function only when combined or        associated in some way with a free morpheme (for example the        negating prefix “un” in undo and the plural suffix “s”).

The field of morphology addresses the structure of morphemes and othertypis of linguistic units such as words, affixes, parts of speech (verb,noun, etc., more formally referred to as “lexical category”),intonation/stress/rhythm (in part more formally referred to as“prosody”), meaning invoked or implied by enveloping context, etc.Morphological analysis also includes a typology framework classifyinglanguages according to the ways by which morphemes are used, forexample:

-   -   Analytic languages that use only isolated (free) morphemes;    -   Agglutinative (“stuck-together”) languages which use bound        morphemes;    -   Fusional languages that use bound morphemes;    -   Polysynthetic languages that form words from groups of many        morphemes (for example the Chukchi word “t        mey        levtp        γt        rk        n” which is composed of eight individual morphemes t-        -mey        -        -levt-peγt-        -rk        n, and more broadly languages allowing for each consonant and        vowel to serve as morphemes.

These examples provide important reference models for options in tactilegestures. For example, in the HDTP approach to touch-based userinterfaces, a gesture can:

-   -   Associate an individual gesteme with an individual morpheme of        general or specific use in an application or group of        applications;    -   Associate a group of two or more este gestemes comprised by a        gesture with an individual morpheme of general or specific use        in an application or group of applications;

Further, a gesture can then be

-   -   Analytic (employing only free morphemes);    -   Agglutinative or Fusional (employing bound morphemes);    -   Polysynthetic (gestures composed of many morphemes.

The invention provides for these and other lexicon constructions to beused in the design and structuring of gestures, gesture meaningstructures, morphemes, gesture lexicon, and gesture grammars.

As an example framework for this, FIG. 30 depicts a representation ofsome correspondences among gestures, gestemes, and the abstractlinguistics concepts of morphemes, words, and sentences.

As an additional example framework for this, FIG. 31a through FIG. 31dprovide finer detail useful in employing additional aspects oftraditional linguistics such as noun phrases, verb phrases, and clausesas is useful for grammatical structure, analysis, and semanticinterpretation.

It is important to note that the HDTP approach to touch-based userinterfaces permits a very wide range of formulations such as thosesuggested above and by other aspects of traditional linguistics. Thatstated, it is equally if not more important to note that the HDTPapproach to touch-based user interfaces does not require inheriting theunnecessary ‘baggage’ of established written or spoken languages (suchas tense matching, noun gender, etc.). Further as to this, typicallyeven the most diverse, robust, and flexible touch-based user interfacewill be used for a range of command/inquiry functions that are far morelimited in scope, nuance, aesthetics, poetics, and so forth than thelanguage of literature, poetry, persuasive discourse, and the like.Thus, in mining what traditional linguistics has to offer, the balancegoal depicted in FIG. 2 is well to be kept in mind.

7.1 Gestural Metaphor, Gestural Onomatopoeia, and Tactile GestureLogography

The HDTP approach to touch-based user interfaces provides for thestructured use of various metaphors in the construction of gestures,strings of gestures, and gestemes. For example, the scope of themetaphor can include:

-   -   The entire gesture, string of gestures, or gesteme;    -   One or more components of a gesture, string of gestures, or        gesteme;    -   One or more aspects of a gesture, string of gestures, or        gesteme.        Additionally, the directness (or degree) of the metaphor can        cover a range such as:    -   Imitative onomatopoeia;    -   Close analogy;    -   Indirect analogy;    -   Analogy of abstractions;    -   Total abstraction.

In traditional linguistics, a logogram is a written character whichrepresents a word or morpheme. Typically a very large number oflogograms are needed to form a general-purpose written language. A greatinterval of time is required to learn the very large number oflogograms. Both these provide a major disadvantage of the logographicsystems over alphabetic systems, but there can be high readingefficiency with logographic writing systems for those who have learnedit. The main logographic system in use today is that of Chinesecharacters. Logographic systems (including written Chinese) includevarious structural and metaphorical elements to aid in associatingmeaning with a given written character within the system.

The HDTP approach to touch-based user interfaces includes provisions forthe gestural equivalent of logograms and logographic systems.

7.2 Appropriate Scope of Gesture Lexicon

The lexicon of a language is comprises its vocabulary. In formallinguistics, lexicon is viewed as a full inventory of the lexemes of thelanguage, where a lexeme is an abstract morphological unit that roughlycorresponds to a set of forms taken by a word (for example “run,”“runs,” “ran,” and “running” are separate distinguished forms of thesame lexeme).

In creating a tactile gesture lexicon, it is likely that the number oflexeme forms can be forced to be one, or else extremely few. Again,typically even the most diverse, robust, and flexible touch-based userinterface will be used for a range of command/inquiry functions that arefar more limited in scope, nuance, aesthetics, poetics, and so forththan the language of literature, poetry, persuasive discourse, and thelike.

7.3 Compound Gestures

Like compound words and word groups that function as a word, the HDTPapproach to touch-based user interfaces provides for individual tactilegestures to be merged by various means to create a new gesture. Examplesof such various means of merger include:

-   -   “Temporally compound” wherein a sequence of two or more tactile        gestures is taken as a composite gesture;    -   “Spatially compound” wherein two or more spatially separated        tactile gestures executed at essentially the same time or        overlapping in time is taken as a composite gesture;    -   “Sequential layering” composition (to be discussed);    -   Geusture forms of portmanteaus wherein two or more gestures or        (gesture-defined morphemes) are combined;    -   Combinations of the two or more instances of one or more of the        above.

Additionally, the HDTP approach to touch-based user interfaces providesfor the use of a systematic system of shortening a string of two or moregestures, for example as in contractions such as “don't,” “it's,” etc.

These tactile examples are not limiting, and the examples and conceptscan be used in other types of user interface systems and other types ofgestures.

7.4 Sequentially-Layered Execution of Gestures

The sequentially-layered execution of tactile gestures can be used tokeep a context throughout a sequence of gestures. Some examplessequentially-layered execution of tactile gestures include:

-   -   Finger 1 performs one or more gestures and stays in place when        completed, then Finger 2 performs one or more gestures, then        end;    -   Finger 1 performs gesture & stays in place when completed, then        Finger 2 performs one or more gestures and stays in place when        completed, then Finger 1 performs one or more gestures, . . . ,        then end;    -   Finger 1 performs gesture & stays in place when completed, then        Finger 2 performs one or more gestures and stays in place when        completed, then Finger 1 performs one or more gestures and stays        in place when completed, then Finger 3 performs one or more        gestures, . . . , then end.    -   Finger 1 performs gesture & stays in place when completed, then        Finger 2 performs one or more gestures and stays in place when        completed, then Finger 3 performs one or more gestures, . . . ,        then end.        Rough representative depictions of the first two examples are        provided respectively as the series FIG. 32a through FIG. 32d        and the series FIG. 33a through FIG. 33 f.

These tactile examples are not limiting, and the examples and conceptscan be used in other types of user interface systems and other types ofgestures.

7.5 Embedded Layering Via Intra-Gestural Prosody Tags

Earlier the notion of “intra-gestural prosody” was introduced throughwhich additional content can be imposed by means of aspects of how atactile gesture is rendered or executed. For example:

-   -   At least one contact angle (yaw, roll, pitch) of the finger(s)        used to render each of the one or more individual strokes        (“gestemes”) making up a tactile gesture;    -   How many fingers used to render each of the one or more        individual strokes (“gestemes”) making up a tactile gesture;    -   Embellishment in individual component element rendering (angle        of rendering, initiating curve, terminating curve,        intra-rendering curve, rates of rendering aspects, etc.);    -   Variations in the relative location of individual gesteme        rendering;    -   What part(s) of the finger or hand used to render each gesteme        of the tactile gesture;    -   Changes in one or more of the above over time.        Intra-gestural prosody can be used as a “tag” to create        additional associations among gestures.

In one use of this, such intra-gestural prosody can be used to createpaths and/or layers of paths. Such paths and layering allows theintroduction of additional material for providing a parallel informationpath, associations, or modifiers.

An example of the use of intra-gestural prosody to render an associationis to use a common but distinguishing intra-gestural prosody—for examplea finger contact angle, or number of fingers—in the rending of two ormore tactile gestures that are to be associated. Another example of theuse of intra-gestural prosody to render an association is to use afinger angle, part of the finger (tip, joint, flat) or number of fingersas a signifier for a particular list item (first items, second item,etc), a particular visual object or range of screen (to the left of thegesture rendering area or cursor, to the right, above, below, 45-degreebelow, nearby, mid-range, far, etc.), a particular element from anoption list (cut, paste, copy, rotate, etc.), and/or other suchapproach.

A simple example of the use of intra-gestural prosody to render amodifier is the rate of change to be used to convey the relative extentof the action represented by the tactile gesture (“verb”)—for examplehow fast and fast to scroll through a list—or the relative extent of thesubject/object (“noun”)—for example how much of the list, text passage,screen region, etc.

8. Phrases, Grammars, and Sentence/Queries

Thus far attention has been largely afforded to the ways individualtactile gestures can be executed, the content and meaning that can beassigned to them, and organizations that can be imposed or used onthese. FIG. 34 depicts an example syntactic and/or semantic hierarchyintegrating the concepts developed thus far.

With such a rich structure, it is entirely possible for two or morealternative gesture sequence expressions to convey the same meaning Thisis suggested in FIG. 35.

The notion of tactile grammars is taught in U.S. Pat. No. 6,570,078,U.S. patent application Ser. Nos. 11/761,978 and 12/418,605, and U.S.Patent Provisional Application 61/449,923. Various broader and moredetailed notions of touch gesture and other gesture linguistics in humanuser interfaces are taught in U.S. patent application Ser. No.12/418,605 and U.S. Patent Provisional Application 61/449,923.

In most computer applications users are either giving commands or makinginquiries (which can be viewed perhaps as a type of command). Examplesinclude:

-   -   “Move—That—Here”;    -   “Copy—That—Here”;    -   “Delete—That”;    -   “Do this—To—That”/“Change—That—This way”;    -   “Create—That—Here”;    -   “What is—That?”    -   “What is (are) the value(s)—of—That?”    -   “Where is—That?”    -   “What is (are)—Objects having that value/value-range/attribute?”

Although Direct Manipulation and WIMP GUIs perhaps reconstitute thesesomewhat in the mind of users as a sequence computer mouse operationsguided by visual feedback, these commands or inquiries are in factnaturally represented as simple sentences. Is this the ultimate fate ofthe potential power and opportunities provide by touch interfaces?

So far today's widely adopted gesture-based multi-touch user interfaceshave added these new time- and labor-saving features:

-   -   Swipe through this 1-dimensional list to this extent;    -   Swipe through this 2-dimensional list at this angle to this        extent;    -   Stretch this image size to this explicit spatial extent;    -   Pinch this image to this explicit spatial extent;    -   Rotate this image by this explicit visual angle;

How much of the capability and opportunities provided by touchinterfaces do these approaches utilize and deliver?

More specifically, as mentioned in the introductory material, the HDTPapproach to touch-based user interfaces provides the basis for:

-   -   (1) a dense, intermixed quantity-rich/symbol-rich/metaphor-rich        information flux capable of significant human-machine        information-transfer rates; and    -   (2) an unprecedented range of natural gestural metaphor support.        The latter (1) and its synergy with the former (2) is especially        noteworthy, emphasized the quote [2]“Gestures are useful for        computer interaction since they are the most primary and        expressive form of human communication.”

So how does technology and industry move forward with gesture-basedinterfaces to a practical, viable next step beyond today's widelyadopted gesture-based multi-touch user interfaces?

-   -   Just broaden the number of built-in Direct Manipulation and WIMP        GUI style manipulation operations than can skip a single menu        step using gesture recognition?    -   Simply add 3D/6D capabilities to map applications, 3D graphics,        games, data visualization, robot arms, etc. and more advanced        menu and color selection functions if the touch interface        provides roll, pitch, raw, and pressure along with X-Y location        and velocity of touch contact?    -   Should the potential and power of touch-based interfaces,        apparently on a scale far closer to spoken and written language        than to that of a computer mouse, be used only for the awkwardly        rendered semantic equivalent of short declarative sentences?

The HDTP approach to touch-based user interfaces in fact provides forsomething far closer to spoken and written language. To explore this,begin with the consideration of some very simple extensions to thesentence representation of traditional Direct Manipulation and WIMP GUIcommands and inquiries listed above into slightly longer sentences. Someexamples might include:

-   -   “Do—This—To Objects having—This value/value-range/attribute”    -   “Apply—This—To Objects previously having—This        value/value-range/attribute”    -   “Find—General objects having that        value/value-range/attribute—Then—Move to—Here”    -   “Find—Graphical objects having that        value/value-range/attribute—Then—Move to—Here—and—Rotate—This        amount”    -   “Find—Physical objects having that        value/value-range/attribute—Then—Move to—Here (2D or 3D        vector)—and—3D—rotate—This amount (vector of angles)”    -   “Find—Physical objects having that        value/value-range/attribute—Then—Move to—Here—In this way        (speed, route, angle)”    -   “Find—Objects having that        value/value-range/attribute—Then—Create—One of these—For        each—Of—Those”

Such very simple extensions are in general exceedingly difficult tosupport using Direct Manipulation and WIMP GUIs, and force users to veryinefficiently break down the desired result into a time-consuming andwrist-fatiguing set of simpler actions that can be handled by DirectManipulation, WIMP GUIs, and today's widely adopted gesture-basedmulti-touch user interfaces.

So yet again consider the quote [2]“Gestures are useful for computerinteraction since they are the most primary and expressive form of humancommunication.” What else is comparable? Speech and traditional writingof course are candidates. What is the raw material of there power oncesymbols (phonetic or orthographic) are formalized? Phrases, grammar,sentences, and higher-level context.

In hopes of leveraging this intrinsic communications machinery, perhapseven directly, attention is now directed to lexical categories, phrasecategories, and context. This permits direct use and then, moreimportantly, extensions to aspects unique to touch interfaces and inparticular the HDTP approach to them.

8.1 Lexical Categories

The invention provides for gestures to be semantically structured asparts of speech (formally termed “lexical categories”) in spoken orwritten languages. Some example lexical categories relevant to commandinterface semantics include:

-   -   Noun;    -   Verb;    -   Adjective;    -   Adverb;    -   Infinitive;    -   Conjunction;    -   Particle.        The invention provides for gestures to be semantically        structured according to and/or including one or more of these        lexical categories, as well as others. Additionally, the        invention provides for at least some gestures to be semantically        structured according to alternative or abstract lexical        categories that are not lexical categories of spoken or written        languages.

8.2 Phrase Categories

The invention provides for such semantically structured gestures to befurther structured according to phrase categories. Example phrasecategories in spoken or written languages include:

-   -   Noun Phrase—noun plus descriptors/modifiers etc that        collectively serves as a noun;    -   Verb Phrase—verb plus descriptors/modifiers etc that        collectively serves as a verb;        Additionally, the invention provides for at least some phrase        categories that are not lexical categories of spoken or written        languages.

8.3 List, Phrase, and Sentence/Query Delimiters

For speech, delimiting between consecutive list items, phrases, andsentences/queries are performed through prosody:

-   -   Temporal pause;    -   Changes in rhythm;    -   Changes in stress;    -   Changes in intonation.        For traditional writing, punctuation is used for delimiting        between consecutive list items, phrases, and sentences/queries:

The HDTP approach to touch-based user interfaces provides for delimitingbetween individual temporal gestures via at least these mechanisms:

-   -   Time separation between two consecutive strings of tactile        gestures;    -   Distance separation between two consecutive strings of        individual tactile gestures;    -   Lexigraphically separation (an tactile gesture string is        unambiguously recognized, and the recognition event invokes a        delineating demarcation between the recognized tactile gesture        string and the next tactile gesture string to follow);    -   Special ending or starting attribute to strings of tactile        gestures;    -   Special delimiting or entry-action gesture(s)—for example        lift-off, tap with another finger, etc.        9. Data Flow Connections Among Tactile Gestures

The invention provides for data flow connections among gestures. Thiscan be accomplished in a number of ways by employing various types ofanalogies from computer and traditional languages, for example:

-   -   Unix™ Pipe standard-input/standard-output chains to define data        flow connections between sequential pairs of tactile gestures        (for example, as depicted in FIG. 36);    -   Traditional linguistic notions of context;    -   Intra-gestural Prosody

As described earlier, other aspects of tactile gestures (for example“intra-gestural prosody”) can be used as modifiers for the gestures.Again, examples of other aspects of tactile gestures include:

-   -   Rate of change of some aspect of a tactile gesture—for example        velocity already in WIMP GUI (cursor location) and today's        widely accepted multi-touch user interfaces (for example, finger        flick velocity affects on scrolling);    -   Interrupted tactile gesture where action is taken by the user        between the rendering of the gestemes comprising the tactile        gesture. To adopt a formal linguistics term, this sort of action        could be called a “gestural endoclitic,” tactile endoclitic” or        “tactile gesture endoclitic;”    -   Contact angles (yaw, roll, pitch);    -   Downward pressure;    -   Additional parameters from multiple finger gestures;    -   Shape parameters (finger-tip, finger-joint, flat-finger, thumb,        etc.).

Recall the example provided earlier of the use of intra-gestural prosodyto render an association through use a shared but distinguishingintra-gestural prosody—for example a finger angle, or number offingers—in the rending of two or more tactile gestures that are to beassociated. This also provides a basis for intra-gestural prosody to beused as to provide data flow connections among tactile gestures.

Also recall the example of the use of intra-gestural prosody to renderan association is to use a finger angle, part of the finger (tip, joint,flat) or number of fingers as a signifier for a particular list item(first items, second item, etc), a particular visual object or range ofscreen (to the left of the gesture rendering area or cursor, to theright, above, below, 45-degree below, nearby, midrange, far, etc.), aparticular element from an option list (cut, paste, copy, rotate, etc.),other such approach. This also provides a basis for intra-gesturalprosody to be used as to provide data flow connections among tactilegestures and/or objects selected by or associated with one or moretactile gestures.

FIG. 37 depicts a representation of an example using intra-gestureprosody as a means of implementing both pipes and other associationsand/or data flow connections.

10. Mapping Tactile Gestures and Actions on Visual-Rendered Objects intoGrammars

The notion of tactile grammars is taught in U.S. Pat. No. 6,570,078,U.S. patent application Ser. Nos. 11/761,978 and 12/418,605, and U.S.Patent Provisional Application 61/449,923.

Various broader and more detailed notions of touch gesture and othergesture linguistics in human user interfaces are taught in U.S. patentapplication Ser. No. 12/418,605 and U.S. Patent Provisional Application61/449,923.

10.1 Parsing Involving Objects that have been Associated with Gestures

Via touchscreen-locating, cursor-location or visually highlighting, atactile gesture can be associated with a visual object rendered on avisual display (or what it is a signifier for, i.e., object, action,etc.). This allows for various types of intuitive primitive grammaticalconstructions. Some examples employing a tactile gesture in forming asubject-verb sentence or inquiry are:

-   -   The underlying (touchscreen), pointed-to (cursor), or selected        (visually highlighted) visual object can serve as a subject noun        and the tactile gesture serve as an operation action verb;    -   The underlying (touchscreen), pointed-to (cursor), or selected        (visually highlighted) visual object can serve as an operation        action verb and the tactile gesture serve as a subject noun;

Some examples employing a spatially-localized tactile gesture in forminga subject-verb-object sentence or inquiry are:

-   -   If context is employed to have earlier in time by some means        selected a subject noun, the underlying (touchscreen),        pointed-to (cursor), or selected (visually highlighted) visual        object can serve as an object noun and the spatially-localized        tactile gesture serve as an operation action verb;    -   If context is employed to have earlier in time by some means        selected a subject noun, the underlying (touchscreen),        pointed-to (cursor), or selected (visually highlighted) visual        object can serve as an operation action verb and the        spatially-localized tactile gesture serve as a object noun;    -   If context is employed to have earlier in time by some means        selected an object noun, the underlying (touchscreen),        pointed-to (cursor), or selected (visually highlighted) visual        object can serve as an subject noun and the spatially-localized        tactile gesture serve as an operation action verb;    -   If context is employed to have earlier in time by some means        selected an object noun, the underlying (touchscreen),        pointed-to (cursor), or selected (visually highlighted) visual        object can serve as an operation action verb, and the        spatially-localized tactile gesture serve as a subject noun;    -   If context is employed to have earlier in time by some means        selected an operation action verb, the underlying (touchscreen),        pointed-to (cursor), or selected (visually highlighted) visual        object can serve as an subject noun, and the spatially-localized        tactile gesture serve as an object noun;    -   If context is employed to have earlier in time by some means        selected an operation action verb, the underlying (touchscreen),        pointed-to (cursor), or selected (visually highlighted) visual        object can serve as an object noun, and the spatially-localized        tactile gesture serve as a subject noun.

Some examples employing a spatially-extended tactile gesture that insome way simultaneously spans two visual objects rendered on a visualdisplay in forming a subject-verb-object sentence or inquiry are:

-   -   One underlying (touchscreen), pointed-to (cursor), or selected        (visually highlighted) visual object can serve as a subject        noun, the other underlying (touchscreen), pointed-to (cursor),        or selected (visually highlighted) visual object can serve as an        object noun and the spatially-extended tactile gesture serve as        an operation action verb;    -   One underlying (touchscreen), pointed-to (cursor), or selected        (visually highlighted) visual object can serve as a subject        noun, the other underlying (touchscreen), pointed-to (cursor),        or selected (visually highlighted) visual object can serve as an        operation action verb, and the spatially-extended tactile        gesture serve as an object noun.

These examples demonstrate how context, order, and spatial-extent ofgestures can be used to map combinations tactile gestures andvisual-rendered objects into grammars; it is thus possible in a similarmanner to include more complex phrase and sentence/inquiryconstructions, for example using gestures and visual-rendered objects,utilizing context, order, and spatial-extent of gestures in variousways, to include:

-   -   Adjectives;    -   Adverbs;    -   Infinitives,    -   Conjunctions and other Particles—for example, “and,” “or,”        negations (“no,” “not”), infinitive markers (“to”), identifier        articles (“the”), conditionals (“unless,” “otherwise”), ordering        (“first”, “second,” “lastly”);    -   Clauses.

Further, as described (at least twice) earlier, other aspects of tactilegestures (for example “intra-gestural prosody”) can be used as modifiersfor the gestures. Again, examples of other aspects of tactile gesturesinclude:

-   -   Rate of change of some aspect of a tactile gesture—for example        velocity already in WIMP GUI (cursor location) and today's        widely accepted multi-touch user interfaces (finger flick        affects on scrolling);    -   Interrupted tactile gesture where action is taken by the user        between the rendering of the gestemes comprising the tactile        gesture;    -   Contact angles (yaw, roll, pitch);    -   Downward pressure;    -   Additional parameters from multiple finger gestures;    -   Shape parameters (finger-tip, finger-joint, flat-finger, thumb,        etc.).

Examples of how the modifiers could be used as an element in a tactilegrammar include:

-   -   Adjective;    -   Adverb;    -   Identifier.

In such an arrangement, such forms intra-gestural prosody can be viewedas a bound morpheme.

10.2 Layered Gesture-Level Metaphors

Mappings between intra-gestural prosody and grammatically-structuredmodifiers provides opportunities for a type of “layered-metaphor” to beused with, incorporated into, or applied to a particular gesture. Forexample:

Lexical Intended Operation Category Tactile Gesture Desired action inthe Verb Metaphorical gesture application How action is performed AdverbIntra-gestural prosody dynamics Attributes of the action AdjectiveIntra-gestural prosody angles. or result pressure, shapes

Of particular note is that a gesture supplemented with intra-gesturalprosody used in this way can function as a noun-phrase, verb-phase, oreven more complex constructions.

FIG. 38 depicts a composite view of some of the key the informationflows supported by the construction provided thus far.

11. Example: Simple Grammars for Rapid Operation of Physical ComputerAided Design (CAD) Systems by HDTP User Interfaces

The following material is adapted from Adapted from U.S. PatentApplication 61/482,248.

Attention is now directed to simple grammars for rapid operation of“physical-model” Computer Aided Design (CAD) systems, for exampleproducts such as Catia™, AutoCAD™, SolidWorks™, Alibre Design™, ViaCAD™,Shark™, and others including specialized 3D CAD systems forarchitecture, plant design, physics modeling, etc.

In such systems, a large number and wide range of operations are used tocreate even the small component elements of a more complex 3D object.For example:

-   -   3D objects of specific primitive shapes are selected and created        in a specified 3D area,    -   Parameters of the shapes of these 3D objects are manipulated,    -   Color and/or texture is applied to the 3D objects    -   The 3D objects are positioned (x,y,z) and oriented (roll, pitch,        yaw) in 3D space    -   The 3D objects are merged with other 3D objects to form        composite 3D objects,    -   The composite 3D objects can be repositioned, reoriented,        resized, reshaped, copied, replicated in specified locations,        etc.

Many of these systems and most of the users who use them perform theseoperations from mouse or mouse-equivalent user interfaces, usuallyallowing only two parameters to be manipulated at a time and involvingthe selection and operation of a large number of palettes, menus,graphical sliders, graphical click buttons, etc. Spatial manipulationsof 3D objects involving three spatial coordinates and three spatialangles, when adjusted two at a time, preclude full-range interactivemanipulations experiences and can create immense combinatorial barriersto positioning and orienting 3D objects in important design phases.Palette and menu selection and manipulations can take many seconds atminimum, and it can often take a minimum of 20 seconds to 2 minutes foran experienced user to create and finalize the simplest primitiveelement.

The HDTP is particularly well suited for 3D CAD and drawing work becauseof both its 3D and 6D capabilities as well as its rich symbol andgrammar capabilities.

FIG. 39a depicts an example of a very simple grammar that can be usedfor rapid control of CAD or drawing software. Here a user first adjustsa finger, plurality of fingers, and/or other part(s) of a hand incontact with an HDTP to cause the adjustment of a generated symbol. Inan example embodiment, the generated symbol can cause a visual responseon a screen. In an embodiment, the visual response can comprise, forexample, one or more of:

-   -   an action on a displayed object,    -   motion of a displayed object,    -   display of text and/or icons,    -   changes in text and/or icons,    -   migration of a highlighting or other effect in a menu, palette,        or 3D arrays,    -   display, changes in, or substitutions of one or more menus,        pallets, or 3D arrays,    -   other outcomes.

In an example embodiment, when the user has selected the desiredcondition, which is equivalent to selection of a particular symbol, thesymbol is then entered. In an example embodiment, the lack ofappreciable motion (i.e., “zero or slow” rate of change) can serve as an“enter” event for the symbol. In another example embodiment, an action(such as a finger tap) can be made by an additional finger, plurality offingers, and/or other part(s) of a hand. These examples are merely meantto be illustrative and is no way limiting and many other variations andalternatives are also possible, anticipated, and provided for by theinvention.

In an example embodiment, after the user has entered the desiredselection (“enter symbol”), the user can then adjust one or more valuesby adjusting a finger, plurality of fingers, and/or other part(s) of ahand in contact with an HDTP. In an embodiment, the visual response cancomprise, for example, one or more of:

-   -   an action on a displayed object,    -   motion of a displayed object,    -   display of text and/or icons,    -   changes in text and/or icons,    -   changes in the state of the object in the CAD or drawing system        software,    -   other outcomes.

This example is merely meant to be illustrative and is no way limitingand many other variations and alternatives are also possible,anticipated, and provided for by the invention.

In an example embodiment, when the user has selected the desired value,the symbol is then entered. In an example embodiment, the lack ofappreciable motion (i.e., “zero or slow” rate of change) can serve as an“enter” event for the value. In another example embodiment, an action(such as a finger tap) can be made by an additional finger, plurality offingers, and/or other part(s) of a hand. These examples are merely meantto be illustrative and is no way limiting and many other variations andalternatives are also possible, anticipated, and provided for by theinvention.

The aforedescribed example sequence and/or other variations can berepeated sequentially, as shown in FIG. 39 b.

Additionally, at least one particular symbol can be used as an “undo” or“re-try” operation. An example of this effect is depicted in FIG. 39 c.

FIG. 40 depicts how the aforedescribed simple grammar can be used tocontrol a CAD or drawing program. In this example, two and/or threefingers (left of the three fingers denoted “1”, middle of the threefingers denoted “2”, right of the three fingers denoted “3”) could beemployed, although many other variations are possible and this exampleis by no means limiting. In one approach, at least finger 2 is used toadjust operations and values, while finger 3 is used to enter theselected symbol or value. Alternatively, the lack of appreciable furthermotion of at least finger 2 can be used to enter the selected symbol orvalue. In FIG. 40, both finger 2 and finger 1 are used to adjustoperations and values. Alternatively, the roles of the fingers in theaforedescribed examples can be exchanges. Alternatively, additionalfingers or other parts of the hand (or two hands) can be used addadditions or substitutions. These examples are merely meant to beillustrative and is no way limiting and many other variations andalternatives are also possible, anticipated, and provided for by theinvention.

As an example of ease of use, the aforedescribed grammar can be used tocreate a shape, modify the shape, position and/or (angularly) orient theshape, and apply a color (as depicted in FIG. 40), all for example in aslittle as a few seconds. In example embodiments of this type, the touchis mostly light and finger motions easy and gentle to execute.

As described earlier, the HDTP and the present invention can support awide range of grammars, including very sophisticated ones. Far moresophisticated grammars can therefore be applied to at least ComputerAided Design (CAD) or drawing software and systems, as well as othersoftware and systems that can benefit from such capabilities.

In an embodiment, an HDTP provides real-time control information toComputer Aided Design (CAD) or drawing software and systems. In anembodiment, an HDTP provides real-time control information to ComputerAided Design (CAD) or drawing software and systems through a USBinterface via HID protocol. In an embodiment, an HDTP provides real-timecontrol information to Computer Aided Design (CAD) or drawing softwareand systems through a HID USB interface abstraction.

The terms “certain embodiments”, “an embodiment”, “embodiment”,“embodiments”, “the embodiment”, “the embodiments”, “one or moreembodiments”, “some embodiments”, and “one embodiment” mean one or more(but not all) embodiments unless expressly specified otherwise. Theterms “including”, “comprising”, “having” and variations thereof mean“including but not limited to”, unless expressly specified otherwise.The enumerated listing of items does not imply that any or all of theitems are mutually exclusive, unless expressly specified otherwise. Theterms “a”, “an” and “the” mean “one or more”, unless expressly specifiedotherwise.

While the invention has been described in detail with reference todisclosed embodiments, various modifications within the scope of theinvention will be apparent to those of ordinary skill in thistechnological field. It is to be appreciated that features describedwith respect to one embodiment typically can be applied to otherembodiments.

The invention can be embodied in other specific forms without departingfrom the spirit or essential characteristics thereof. The presentembodiments are therefore to be considered in all respects asillustrative and not restrictive, the scope of the invention beingindicated by the appended claims rather than by the foregoingdescription, and all changes which come within the meaning and range ofequivalency of the claims are therefore intended to be embraced therein.

Although exemplary embodiments have been provided in detail, variouschanges, substitutions and alternations could be made thereto withoutdeparting from spirit and scope of the disclosed subject matter asdefined by the appended claims. Variations described for the embodimentsmay be realized in any combination desirable for each particularapplication. Thus particular limitations and embodiment enhancementsdescribed herein, which may have particular advantages to a particularapplication, need not be used for all applications. Also, not alllimitations need be implemented in methods, systems, and apparatusesincluding one or more concepts described with relation to the providedembodiments. Therefore, the invention properly is to be construed withreference to the claims.

REFERENCES

-   [1] Shneiderman “Direct Manipulation. A Step Beyond Programming    Languages” IEEE Transactions on Computers 16 (8), 1983, pp. 57-69.-   [2] J. Wachs, M. Kolsch, H. Stern, Y. Edan, “Vision-Based    Hand-Gesture Applications,” Communications of the ACM, Vol. 54 No.    3, February 2011, pp. 60-71.-   [3] M. Eden, “On the Formalization of Handwriting,” in Structure of    Language and its Mathematical Aspects, American Mathematical    Society, 1961.

The invention claimed is:
 1. A method for a multi-touch gesture- baseduser interface, the method comprising: concatenating a first subset ofgestemes from a plurality of gestemes based on at least temporal logicand at least one parameter associated with a space function forconstructing a first gesture, each gesteme being primitive gesturesegments; concatenating a second subset of gestemes from the pluralityof gestemes based on at least function for temporal logic and at leastone parameter associated with the space function for constructing asecond gesture; receiving real-time multi-touch gesture-basedinformation from a multi-touch gesture-based user interface; processingthe real-time multi-touch gesture-based information to sequentiallyidentify over intervals of real-time at least a recognized sequence ofspecific gestemes belonging to the plurality of gestemes; determiningfrom the sequence of gestemes that the user's execution of at least onegesture has been completed; determining from the sequence of gestemes aspecific gesture represented by the sequence of gestemes; wherein thegesture is recognized as either the first gesture or the second gestureaccording to the sequence of gestemes.
 2. The method of claim 1 whereingesture grammars are provided using the sequence of gestemes.
 3. Themethod of claim 1 wherein structured-meaning gesture- lexicon frameworksare provided using the sequence of gestemes.
 4. The method of claim 1wherein gesture context frameworks are provided using the sequence ofgestemes.
 5. The method of claim 1 wherein the use of gesture prosody isprovided.
 6. A method for controlling an application operating on aprocessor with a multi-touch gesture-based user interface, the methodcomprising: receiving a gesture in real time from a high-definitiontouch pad (HDTP); deconstructing the gesture into a sequence of gestemesas the gesture is received, each gesteme comprising distinct primitivegesture segments; determining from the sequence of gestemes that thegesture has been completed; recognizing the sequence of gesteme as aparticular command to the application; and transmitting the particularcommand to the application operating on the processor.
 7. The method ofclaim 6 wherein gesture grammars are provided using the sequence ofgestemes.
 8. The method of claim 6 wherein structured-meaninggesture-lexicon frameworks are provided using the sequence of gestemes.9. The method of claim 6 wherein gesture context frameworks are providedusing the sequence of gestemes.
 10. The method of claim 6 wherein theuse of gesture prosody is provided.